Repository: tidyverse/ellmer
Pull request #496: First pass at `batch_chat()` implementation
Author: login: hadley
Created: 2025-05-14T21:40:09Z | Merged: 2025-05-16T15:14:01Z

Focussing first on Anthropic + OpenAI. Part of #143.

To do:

* [x] Figure out some testing strategy. At least test the flow with some mocked methods?
* [x] Implement `batch_chat_structured()`
* [x] Think about if we want to store a hash of the chat + prompts to warn the user if there's a mismatch
* [x] Better progress bar

@gadenbuie While the implementation isn't 100% complete (and I haven't written any tests yet), I'd love to get your thoughts on this API. In particular, what do you think of the required `path` argument, and is it correct to throw an error if `wait = FALSE`.


Diff:
diff --git a/DESCRIPTION b/DESCRIPTION
index b50987bd3..6179bbb2a 100644
--- a/DESCRIPTION
+++ b/DESCRIPTION
@@ -61,6 +61,7 @@ Collate:
     'content.R'
     'provider.R'
     'as-json.R'
+    'batch-chat.R'
     'chat-parallel.R'
     'chat-structured.R'
     'utils-coro.R'
@@ -103,5 +104,6 @@ Collate:
     'utils-callbacks.R'
     'utils-cat.R'
     'utils-merge.R'
+    'utils-prettytime.R'
     'utils.R'
     'zzz.R'
diff --git a/NAMESPACE b/NAMESPACE
index 2bac880b6..224b949e1 100644
--- a/NAMESPACE
+++ b/NAMESPACE
@@ -20,6 +20,9 @@ export(TypeArray)
 export(TypeBasic)
 export(TypeEnum)
 export(TypeObject)
+export(batch_chat)
+export(batch_chat_completed)
+export(batch_chat_structured)
 export(chat_anthropic)
 export(chat_aws_bedrock)
 export(chat_azure)
diff --git a/NEWS.md b/NEWS.md
index a2680120c..55359a51b 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,5 +1,9 @@
 # ellmer (development version)
 
+* `batch_chat()` allows you to submit multiple chats to OpenAI and Anthropic's
+  batched interfaces. These only guarantee a response within 24 hours, but are
+  50% of the price of regular requests (#143).
+
 * `chat_ollama()` now accepts `api_key` or consults the `OLLAMA_API_KEY`
   environment variable. This is not needed for local usage, but enables
   bearer-token authentication when Ollama is running behind a reverse proxy
diff --git a/R/batch-chat.R b/R/batch-chat.R
new file mode 100644
index 000000000..bdc48f9b1
--- /dev/null
+++ b/R/batch-chat.R
@@ -0,0 +1,339 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` and `batch_chat_structured()` currently only work with
+#' [chat_openai()] and [chat_anthropic()]. They use the
+#' [OpenAI](https://platform.openai.com/docs/guides/batch) and
+#' [Anthropic](https://docs.anthropic.com/en/docs/build-with-claude/batch-processing)
+#' batch APIs which allow you to submit multiple requests simultaneously.
+#' The results can take up to 24 hours to complete, but in return you pay 50%
+#' less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, or
+#' you're working with a different provider, you may want to use
+#' [parallel_chat()] instead.
+#'
+#' Since batched requests can take a long time to complete, `batch_chat()`
+#' requires a file path that is used to store information about the batch so
+#' you never lose any work. You can either set `wait = FALSE` or simply
+#' interrupt the waiting process, then later, either call `batch_chat()` to
+#' resume where you left off or call `batch_chat_completed()` to see if the
+#' results are ready to retrieve. `batch_chat()` will store the chat responses
+#' in this file, so you can either keep it around to cache the results,
+#' or delete it to free up disk space.
+#'
+#' This API is marked as experimental since I don't yet know how to handle
+#' errors in the most helpful way. Fortunately they don't seem to be common,
+#' but if you have ideas, please let me know!
+#'
+#' @inheritParams parallel_chat
+#' @param path Path to file (with `.json` extension) to store state.
+#'
+#'   The file records a hash of the provider, the prompts, and the existing
+#'   chat turns. If you attempt to reuse the same file with any of these being
+#'   different, you'll get an error.
+#' @param wait If `TRUE`, will wait for batch to complete. If `FALSE`,
+#'   it will return `NULL` if the batch is not complete, and you can retrieve
+#'   the results later by re-running `batch_chat()` when
+#'   `batch_chat_completed()` is `TRUE`.
+#' @examplesIf has_credentials("openai")
+#' chat <- chat_openai(model = "gpt-4.1-nano")
+#'
+#' # Chat ----------------------------------------------------------------------
+#'
+#' prompts <- interpolate("What do people from {{state.name}} bring to a potluck dinner?")
+#' \dontrun{
+#' chats <- batch_chat(chat, prompts, path = "potluck.json")
+#' chats
+#' }
+#'
+#' # Structured data -----------------------------------------------------------
+#' prompts <- list(
+#'   "I go by Alex. 42 years on this planet and counting.",
+#'   "Pleased to meet you! I'm Jamal, age 27.",
+#'   "They call me Li Wei. Nineteen years young.",
+#'   "Fatima here. Just celebrated my 35th birthday last week.",
+#'   "The name's Robert - 51 years old and proud of it.",
+#'   "Kwame here - just hit the big 5-0 this year."
+#' )
+#' type_person <- type_object(name = type_string(), age = type_number())
+#' \dontrun{
+#' data <- batch_chat_structured(
+#'   chat = chat,
+#'   prompts = prompts,
+#'   path = "people-data.json",
+#'   type = type_person
+#' )
+#' data
+#' }
+#'
+#' @export
+batch_chat <- function(chat, prompts, path, wait = TRUE) {
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    path = path,
+    wait = wait
+  )
+  job$step_until_done()
+
+  assistant_turns <- job$result_turns()
+  map2(job$user_turns, assistant_turns, function(user, assistant) {
+    if (!is.null(assistant)) {
+      chat$clone()$add_turn(user, assistant)
+    } else {
+      NULL
+    }
+  })
+}
+
+#' @export
+#' @rdname batch_chat
+#' @inheritParams parallel_chat_structured
+batch_chat_structured <- function(
+  chat,
+  prompts,
+  path,
+  type,
+  wait = TRUE,
+  convert = TRUE,
+  include_tokens = FALSE,
+  include_cost = FALSE
+) {
+  check_chat(chat)
+  provider <- chat$get_provider()
+  needs_wrapper <- S7_inherits(provider, ProviderOpenAI)
+
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    type = wrap_type_if_needed(type, needs_wrapper),
+    path = path,
+    wait = wait,
+    call = error_call
+  )
+  job$step_until_done()
+  turns <- job$result_turns()
+
+  multi_convert(
+    provider,
+    turns,
+    type,
+    convert = convert,
+    include_tokens = include_tokens,
+    include_cost = include_cost
+  )
+}
+
+#' @export
+#' @rdname batch_chat
+batch_chat_completed <- function(chat, prompts, path) {
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    path = path
+  )
+  switch(
+    job$stage,
+    "submitting" = FALSE,
+    "waiting" = !job$poll()$working,
+    "retrieving" = TRUE,
+    "done" = TRUE,
+    cli::cli_abort("Unexpected stage: {job$stage}", .internal = TRUE)
+  )
+}
+
+BatchJob <- R6::R6Class(
+  "BatchJob",
+  public = list(
+    chat = NULL,
+    user_turns = NULL,
+    path = NULL,
+    should_wait = TRUE,
+    type = NULL,
+
+    # Internal state
+    provider = NULL,
+    started_at = NULL,
+    stage = NULL,
+    batch = NULL,
+    results = NULL,
+
+    initialize = function(
+      chat,
+      prompts,
+      path,
+      type = NULL,
+      wait = TRUE,
+      call = caller_env(2)
+    ) {
+      check_chat(chat, call = call)
+      self$provider <- chat$get_provider()
+      check_has_batch_support(self$provider, call = call)
+
+      user_turns <- as_user_turns(prompts, call = call)
+      check_string(path, allow_empty = FALSE, call = call)
+      check_bool(wait, call = call)
+
+      self$chat <- chat
+      self$user_turns <- user_turns
+      self$type <- type
+      self$path <- path
+      self$should_wait <- wait
+
+      if (file.exists(path)) {
+        state <- jsonlite::read_json(path)
+        self$stage <- state$stage
+        self$batch <- state$batch
+        self$results <- state$results
+        self$started_at <- .POSIXct(state$started_at)
+
+        self$check_hash(state$hash, call = call)
+      } else {
+        self$stage <- "submitting"
+        self$batch <- NULL
+        self$started_at <- Sys.time()
+      }
+    },
+
+    save_state = function() {
+      jsonlite::write_json(
+        list(
+          version = 1,
+          stage = self$stage,
+          batch = self$batch,
+          results = self$results,
+          started_at = as.integer(self$started_at),
+          hash = self$compute_hash()
+        ),
+        self$path,
+        auto_unbox = TRUE,
+        pretty = TRUE
+      )
+    },
+
+    step = function() {
+      if (self$stage == "submitting") {
+        self$submit()
+      } else if (self$stage == "waiting") {
+        self$wait()
+      } else if (self$stage == "retrieving") {
+        self$retrieve()
+      } else {
+        cli::cli_abort("Unknown stage: {self$stage}", .internal = TRUE)
+      }
+    },
+
+    step_until_done = function() {
+      while (self$stage != "done") {
+        if (!self$step()) {
+          return(invisible())
+        }
+      }
+      invisible(self)
+    },
+
+    submit = function() {
+      existing <- self$chat$get_turns(include_system_prompt = TRUE)
+      conversations <- append_turns(list(existing), self$user_turns)
+
+      self$batch <- batch_submit(self$provider, conversations, type = self$type)
+      self$stage <- "waiting"
+      self$save_state()
+      TRUE
+    },
+
+    wait = function() {
+      # always poll once, even when wait = FALSE
+      status <- self$poll()
+
+      if (self$should_wait) {
+        cli::cli_progress_bar(
+          format = paste(
+            "{cli::pb_spin} Processing...",
+            "[{self$elapsed()}]",
+            "{status$n_processing} pending |",
+            "{cli::col_green({status$n_succeeded})} done |",
+            "{cli::col_red({status$n_failed})} failed"
+          )
+        )
+        while (status$working) {
+          Sys.sleep(0.5)
+          cli::cli_progress_update()
+          status <- self$poll()
+        }
+        cli::cli_progress_done()
+      }
+
+      if (!status$working) {
+        self$stage <- "retrieving"
+        self$save_state()
+        TRUE
+      } else {
+        FALSE
+      }
+    },
+    poll = function() {
+      self$batch <- batch_poll(self$provider, self$batch)
+      self$save_state()
+
+      batch_status(self$provider, self$batch)
+    },
+    elapsed = function() {
+      pretty_sec(as.integer(Sys.time()) - as.integer(self$started_at))
+    },
+
+    retrieve = function() {
+      self$results <- batch_retrieve(self$provider, self$batch)
+      self$stage <- "done"
+      self$save_state()
+      TRUE
+    },
+
+    result_turns = function() {
+      map2(self$results, self$user_turns, function(result, user_turn) {
+        batch_result_turn(self$provider, result, has_type = !is.null(self$type))
+      })
+    },
+
+    compute_hash = function() {
+      # TODO: replace with JSON serialization when available
+      list(
+        provider = hash(props(self$provider)),
+        prompts = hash(lapply(self$user_turns, format)),
+        user_turns = hash(lapply(self$chat$get_turns(TRUE), format))
+      )
+    },
+
+    check_hash = function(old_hash, call = caller_env()) {
+      new_hash <- self$compute_hash()
+      same <- map2_lgl(old_hash, new_hash, `==`)
+
+      if (all(same)) {
+        return(invisible())
+      }
+      differences <- names(new_hash)[!same]
+
+      cli::cli_abort(
+        c(
+          "{differences} don't match stored values.",
+          i = "Do you need to pick a different {.arg path}?"
+        ),
+        call = call
+      )
+    }
+  )
+)
+
+
+check_has_batch_support <- function(provider, call = caller_env()) {
+  if (has_batch_support(provider)) {
+    return(invisible())
+  }
+
+  cli::cli_abort(
+    "Batch requests are not currently supported by this provider.",
+    call = call
+  )
+}
diff --git a/R/chat-parallel.R b/R/chat-parallel.R
index 7e1328c72..16758bc03 100644
--- a/R/chat-parallel.R
+++ b/R/chat-parallel.R
@@ -7,6 +7,10 @@
 #' typically considerably faster than submitting them in sequence, especially
 #' with Gemini and OpenAI.
 #'
+#' If you're using [chat_openai()] or [chat_anthropic()] and you're willing
+#' to wait longer, you might want to use [batch_chat()] instead, as it comes
+#' with a 50% discount in return for taking up to 24 hours.
+#'
 #' @param chat A base chat object.
 #' @param prompts A vector created by [interpolate()] or a list
 #'   of character vectors.
@@ -47,6 +51,7 @@
 #' type_person <- type_object(name = type_string(), age = type_number())
 #' parallel_chat_structured(chat, prompts, type_person)
 parallel_chat <- function(chat, prompts, max_active = 10, rpm = 500) {
+  check_chat(chat)
   my_parallel_turns <- function(conversations) {
     parallel_turns(
       provider = chat$get_provider(),
@@ -120,7 +125,6 @@ parallel_chat_structured <- function(
 
   provider <- chat$get_provider()
   needs_wrapper <- S7_inherits(provider, ProviderOpenAI)
-  w_type <- wrap_type_if_needed(type, needs_wrapper)
 
   # First build up list of cumulative conversations
   user_turns <- as_user_turns(prompts)
@@ -131,13 +135,38 @@ parallel_chat_structured <- function(
     provider = provider,
     conversations = conversations,
     tools = chat$get_tools(),
-    type = w_type,
+    type = wrap_type_if_needed(type, needs_wrapper),
     max_active = max_active,
     rpm = rpm
   )
 
+  multi_convert(
+    provider,
+    turns,
+    type,
+    convert = convert,
+    include_tokens = include_tokens,
+    include_cost = include_cost
+  )
+}
+
+multi_convert <- function(
+  provider,
+  turns,
+  type,
+  convert = TRUE,
+  include_tokens = FALSE,
+  include_cost = FALSE
+) {
+  needs_wrapper <- S7_inherits(provider, ProviderOpenAI)
+
   rows <- map(turns, \(turn) {
-    extract_data(turn, w_type, convert = FALSE, needs_wrapper = needs_wrapper)
+    extract_data(
+      turn = turn,
+      type = wrap_type_if_needed(type, needs_wrapper),
+      convert = FALSE,
+      needs_wrapper = needs_wrapper
+    )
   })
 
   if (convert) {
diff --git a/R/chat.R b/R/chat.R
index 47724d361..88d89facb 100644
--- a/R/chat.R
+++ b/R/chat.R
@@ -793,6 +793,15 @@ is_chat <- function(x) {
   inherits(x, "Chat")
 }
 
+
+check_chat <- function(chat, call = caller_env()) {
+  if (is_chat(chat)) {
+    return(invisible())
+  }
+
+  cli::cli_abort("{.arg chat} must be a <Chat> object.", call = call)
+}
+
 #' @export
 print.Chat <- function(x, ...) {
   provider <- x$get_provider()
diff --git a/R/provider-claude.R b/R/provider-claude.R
index 62cbcf762..4f791b867 100644
--- a/R/provider-claude.R
+++ b/R/provider-claude.R
@@ -406,6 +406,88 @@ method(as_json, list(ProviderAnthropic, ContentThinking)) <- function(
     signature = x@extra$signature
   )
 }
+
+# Batch chat -------------------------------------------------------------------
+
+method(has_batch_support, ProviderAnthropic) <- function(provider) {
+  TRUE
+}
+
+# https://docs.anthropic.com/en/api/creating-message-batches
+method(batch_submit, ProviderAnthropic) <- function(
+  provider,
+  conversations,
+  type = NULL
+) {
+  req <- base_request(provider)
+  req <- req_url_path_append(req, "/messages/batches")
+
+  requests <- map(seq_along(conversations), function(i) {
+    params <- chat_body(
+      provider,
+      stream = FALSE,
+      turns = conversations[[i]],
+      type = type
+    )
+    list(
+      custom_id = paste0("chat-", i),
+      params = params
+    )
+  })
+  req <- req_body_json(req, list(requests = requests))
+
+  resp <- req_perform(req)
+  resp_body_json(resp)
+}
+
+# https://docs.anthropic.com/en/api/retrieving-message-batches
+method(batch_poll, ProviderAnthropic) <- function(provider, batch) {
+  req <- base_request(provider)
+  req <- req_url_path_append(req, "/messages/batches/", batch$id)
+  resp <- req_perform(req)
+
+  resp_body_json(resp)
+}
+
+method(batch_status, ProviderAnthropic) <- function(provider, batch) {
+  counts <- batch$request_counts
+  list(
+    working = batch$processing_status != "ended",
+    n_processing = batch$request_counts$processing,
+    n_succeeded = batch$request_counts$succeeded,
+    n_failed = counts$errored + counts$canceled + counts$expired
+  )
+}
+
+# https://docs.anthropic.com/en/api/retrieving-message-batch-results
+method(batch_retrieve, ProviderAnthropic) <- function(provider, batch) {
+  req <- base_request(provider)
+  req <- req_url(req, batch$results_url)
+  req <- req_progress(req, "down")
+
+  path <- withr::local_tempfile()
+  req <- req_perform(req, path = path)
+
+  lines <- readLines(path, warn = FALSE)
+  json <- lapply(lines, jsonlite::fromJSON, simplifyVector = FALSE)
+
+  ids <- as.numeric(gsub("chat-", "", map_chr(json, "[[", "custom_id")))
+  results <- lapply(json, "[[", "result")
+  results[order(ids)]
+}
+
+method(batch_result_turn, ProviderAnthropic) <- function(
+  provider,
+  result,
+  has_type = FALSE
+) {
+  if (result$type == "succeeded") {
+    value_turn(provider, result$message, has_type = has_type)
+  } else {
+    NULL
+  }
+}
+
 # Pricing ----------------------------------------------------------------------
 
 method(standardise_model, ProviderAnthropic) <- function(provider, model) {
diff --git a/R/provider-openai.R b/R/provider-openai.R
index 3770f07d4..36a3bc849 100644
--- a/R/provider-openai.R
+++ b/R/provider-openai.R
@@ -388,6 +388,122 @@ method(as_json, list(ProviderOpenAI, TypeObject)) <- function(provider, x) {
   )
 }
 
+
+# Batched requests -------------------------------------------------------------
+
+method(has_batch_support, ProviderOpenAI) <- function(provider) {
+  # Only enable for OpenAI, not subclasses
+  provider@name == "OpenAI"
+}
+
+# https://platform.openai.com/docs/api-reference/batch
+method(batch_submit, ProviderOpenAI) <- function(
+  provider,
+  conversations,
+  type = NULL
+) {
+  path <- withr::local_tempfile()
+
+  # First put the requests in a file
+  # https://platform.openai.com/docs/api-reference/batch/request-input
+  requests <- map(seq_along(conversations), function(i) {
+    body <- chat_body(
+      provider,
+      stream = FALSE,
+      turns = conversations[[i]],
+      type = type
+    )
+
+    list(
+      custom_id = paste0("chat-", i),
+      method = "POST",
+      url = "/v1/chat/completions",
+      body = body
+    )
+  })
+  json <- map_chr(requests, jsonlite::toJSON, auto_unbox = TRUE)
+  writeLines(json, path)
+  # Then upload it
+  uploaded <- openai_upload(provider, path)
+
+  # Now we can submit the
+  req <- base_request(provider)
+  req <- req_url_path_append(req, "/batches")
+  req <- req_body_json(
+    req,
+    list(
+      input_file_id = uploaded$id,
+      endpoint = "/v1/chat/completions",
+      completion_window = "24h"
+    )
+  )
+
+  resp <- req_perform(req)
+  resp_body_json(resp)
+}
+
+# https://platform.openai.com/docs/api-reference/batch/retrieve
+openai_upload <- function(provider, path, purpose = "batch") {
+  req <- base_request(provider)
+  req <- req_url_path_append(req, "/files")
+  req <- req_body_multipart(
+    req,
+    purpose = purpose,
+    file = curl::form_file(path)
+  )
+  req <- req_progress(req, "up")
+
+  resp <- req_perform(req)
+  resp_body_json(resp)
+}
+
+# https://docs.anthropic.com/en/api/retrieving-message-batches
+method(batch_poll, ProviderOpenAI) <- function(provider, batch) {
+  req <- base_request(provider)
+  req <- req_url_path_append(req, "/batches/", batch$id)
+
+  resp <- req_perform(req)
+  resp_body_json(resp)
+}
+method(batch_status, ProviderOpenAI) <- function(provider, batch) {
+  list(
+    working = batch$status != "completed",
+    n_processing = batch$request_counts$total - batch$request_counts$completed,
+    n_succeeded = batch$request_counts$completed,
+    n_failed = batch$request_counts$failed
+  )
+}
+
+
+# https://docs.anthropic.com/en/api/retrieving-message-batch-results
+method(batch_retrieve, ProviderOpenAI) <- function(provider, batch) {
+  path <- withr::local_tempfile()
+
+  req <- base_request(provider)
+  req <- req_url_path_append(req, "/files/", batch$output_file_id, "/content")
+  req <- req_progress(req, "down")
+  resp <- req_perform(req, path = path)
+
+  lines <- readLines(path, warn = FALSE)
+  json <- lapply(lines, jsonlite::fromJSON, simplifyVector = FALSE)
+
+  ids <- as.numeric(gsub("chat-", "", map_chr(json, "[[", "custom_id")))
+  results <- lapply(json, "[[", "response")
+  results[order(ids)]
+}
+
+method(batch_result_turn, ProviderOpenAI) <- function(
+  provider,
+  result,
+  has_type = FALSE
+) {
+  if (result$status_code == 200) {
+    value_turn(provider, result$body, has_type = has_type)
+  } else {
+    NULL
+  }
+}
+
 # Models -----------------------------------------------------------------------
 
 #' @rdname chat_openai
diff --git a/R/provider.R b/R/provider.R
index 7f8cbd4ad..f9b3a92f2 100644
--- a/R/provider.R
+++ b/R/provider.R
@@ -167,6 +167,71 @@ method(as_json, list(Provider, ContentJson)) <- function(provider, x) {
   as_json(provider, ContentText("<structured data/>"))
 }
 
+# Batch AI ---------------------------------------------------------------
+
+# Does the provider support batch uploads?
+has_batch_support <- new_generic(
+  "has_batch_support",
+  "provider",
+  function(provider) {
+    S7_dispatch()
+  }
+)
+method(has_batch_support, Provider) <- function(provider) {
+  FALSE
+}
+
+# Submit a batch, return an object "batch" object that will be passed to
+# batch_poll() and batch_retrieve()
+batch_submit <- new_generic(
+  "batch_submit",
+  "provider",
+  function(provider, conversations, type = NULL) {
+    S7_dispatch()
+  }
+)
+
+# Get batch status. Returns an opaque list.
+batch_poll <- new_generic(
+  "batch_poll",
+  "provider",
+  function(provider, batch) {
+    S7_dispatch()
+  }
+)
+
+# Given batch status, return a standardised list:
+# * working - TRUE/FALSE
+# * n_processing = number of requests still processing
+# * n_succeeded = number of requests that succeeded
+# * n_failed = number of requests that failed
+batch_status <- new_generic(
+  "batch_status",
+  "provider",
+  function(provider, batch) {
+    S7_dispatch()
+  }
+)
+
+# Download batched results
+batch_retrieve <- new_generic(
+  "batch_retrieve",
+  "provider",
+  function(provider, batch) {
+    S7_dispatch()
+  }
+)
+
+# Process a single result. Returns either a turn or NULL, if the turn
+# did not succeed
+batch_result_turn <- new_generic(
+  "batch_result_turn",
+  "provider",
+  function(provider, result, has_type = FALSE) {
+    S7_dispatch()
+  }
+)
+
 # Pricing ---------------------------------------------------------------------
 
 standardise_model <- new_generic(
diff --git a/R/utils-prettytime.R b/R/utils-prettytime.R
new file mode 100644
index 000000000..da9f050ba
--- /dev/null
+++ b/R/utils-prettytime.R
@@ -0,0 +1,62 @@
+parse_ms <- function(ms) {
+  stopifnot(is.numeric(ms))
+
+  data.frame(
+    days = floor(ms / 86400000),
+    hours = floor((ms / 3600000) %% 24),
+    minutes = floor((ms / 60000) %% 60),
+    seconds = round((ms / 1000) %% 60, 1)
+  )
+}
+
+first_positive <- function(x) which(x > 0)[1]
+
+trim <- function(x) gsub("^\\s+|\\s+$", "", x)
+
+pretty_ms <- function(ms, compact = FALSE) {
+  stopifnot(is.numeric(ms))
+
+  parsed <- t(parse_ms(ms))
+
+  if (compact) {
+    units <- c("d", "h", "m", "s")
+    parsed2 <- parsed
+    parsed2[] <- paste0(parsed, units)
+    idx <- cbind(
+      apply(parsed, 2, first_positive),
+      seq_len(length(ms))
+    )
+    tmp <- paste0("~", parsed2[idx])
+
+    # handle NAs
+    tmp[is.na(parsed2[idx])] <- NA_character_
+    tmp
+  } else {
+    ## Exact for small ones
+    exact <- paste0(ceiling(ms), "ms")
+    exact[is.na(ms)] <- NA_character_
+
+    ## Approximate for others, in seconds
+    merge_pieces <- function(pieces) {
+      ## handle NAs
+      if (all(is.na(pieces))) {
+        return(NA_character_)
+      }
+
+      ## handle non-NAs
+      paste0(
+        if (pieces[1]) paste0(pieces[1], "d "),
+        if (pieces[2]) paste0(pieces[2], "h "),
+        if (pieces[3]) paste0(pieces[3], "m "),
+        if (pieces[4]) paste0(pieces[4], "s ")
+      )
+    }
+    approx <- trim(apply(parsed, 2, merge_pieces))
+
+    ifelse(ms < 1000, exact, approx)
+  }
+}
+
+pretty_sec <- function(sec, compact = FALSE) {
+  pretty_ms(sec * 1000, compact = compact)
+}
diff --git a/_pkgdown.yml b/_pkgdown.yml
index 8f6a1114e..ad52a9025 100644
--- a/_pkgdown.yml
+++ b/_pkgdown.yml
@@ -33,6 +33,7 @@ reference:
 
   - title: Parallel and batch chat
     contents:
+      - batch_chat
       - parallel_chat
 
   - title: Tools and structured data
diff --git a/man/batch_chat.Rd b/man/batch_chat.Rd
new file mode 100644
index 000000000..a6c2bc6ba
--- /dev/null
+++ b/man/batch_chat.Rd
@@ -0,0 +1,115 @@
+% Generated by roxygen2: do not edit by hand
+% Please edit documentation in R/batch-chat.R
+\name{batch_chat}
+\alias{batch_chat}
+\alias{batch_chat_structured}
+\alias{batch_chat_completed}
+\title{Submit multiple chats in one batch}
+\usage{
+batch_chat(chat, prompts, path, wait = TRUE)
+
+batch_chat_structured(
+  chat,
+  prompts,
+  path,
+  type,
+  wait = TRUE,
+  convert = TRUE,
+  include_tokens = FALSE,
+  include_cost = FALSE
+)
+
+batch_chat_completed(chat, prompts, path)
+}
+\arguments{
+\item{chat}{A base chat object.}
+
+\item{prompts}{A vector created by \code{\link[=interpolate]{interpolate()}} or a list
+of character vectors.}
+
+\item{path}{Path to file (with \code{.json} extension) to store state.
+
+The file records a hash of the provider, the prompts, and the existing
+chat turns. If you attempt to reuse the same file with any of these being
+different, you'll get an error.}
+
+\item{wait}{If \code{TRUE}, will wait for batch to complete. If \code{FALSE},
+it will return \code{NULL} if the batch is not complete, and you can retrieve
+the results later by re-running \code{batch_chat()} when
+\code{batch_chat_completed()} is \code{TRUE}.}
+
+\item{type}{A type specification for the extracted data. Should be
+created with a \code{\link[=type_boolean]{type_()}} function.}
+
+\item{convert}{If \code{TRUE}, automatically convert from JSON lists to R
+data types using the schema. This typically works best when \code{type} is
+\code{\link[=type_object]{type_object()}} as this will give you a data frame with one column for
+each property. If \code{FALSE}, returns a list.}
+
+\item{include_tokens}{If \code{TRUE}, and the result is a data frame, will
+add \code{input_tokens} and \code{output_tokens} columns giving the total input
+and output tokens for each prompt.}
+
+\item{include_cost}{If \code{TRUE}, and the result is a data frame, will
+add \code{cost} column giving the cost of each prompt.}
+}
+\description{
+\ifelse{html}{\href{https://lifecycle.r-lib.org/articles/stages.html#experimental}{\figure{lifecycle-experimental.svg}{options: alt='[Experimental]'}}}{\strong{[Experimental]}}
+
+\code{batch_chat()} and \code{batch_chat_structured()} currently only work with
+\code{\link[=chat_openai]{chat_openai()}} and \code{\link[=chat_anthropic]{chat_anthropic()}}. They use the
+\href{https://platform.openai.com/docs/guides/batch}{OpenAI} and
+\href{https://docs.anthropic.com/en/docs/build-with-claude/batch-processing}{Anthropic}
+batch APIs which allow you to submit multiple requests simultaneously.
+The results can take up to 24 hours to complete, but in return you pay 50\%
+less than usual (but note that ellmer doesn't include this discount in
+its pricing metadata). If you want to get results back more quickly, or
+you're working with a different provider, you may want to use
+\code{\link[=parallel_chat]{parallel_chat()}} instead.
+
+Since batched requests can take a long time to complete, \code{batch_chat()}
+requires a file path that is used to store information about the batch so
+you never lose any work. You can either set \code{wait = FALSE} or simply
+interrupt the waiting process, then later, either call \code{batch_chat()} to
+resume where you left off or call \code{batch_chat_completed()} to see if the
+results are ready to retrieve. \code{batch_chat()} will store the chat responses
+in this file, so you can either keep it around to cache the results,
+or delete it to free up disk space.
+
+This API is marked as experimental since I don't yet know how to handle
+errors in the most helpful way. Fortunately they don't seem to be common,
+but if you have ideas, please let me know!
+}
+\examples{
+\dontshow{if (has_credentials("openai")) (if (getRversion() >= "3.4") withAutoprint else force)(\{ # examplesIf}
+chat <- chat_openai(model = "gpt-4.1-nano")
+
+# Chat ----------------------------------------------------------------------
+
+prompts <- interpolate("What do people from {{state.name}} bring to a potluck dinner?")
+\dontrun{
+chats <- batch_chat(chat, prompts, path = "potluck.json")
+chats
+}
+
+# Structured data -----------------------------------------------------------
+prompts <- list(
+  "I go by Alex. 42 years on this planet and counting.",
+  "Pleased to meet you! I'm Jamal, age 27.",
+  "They call me Li Wei. Nineteen years young.",
+  "Fatima here. Just celebrated my 35th birthday last week.",
+  "The name's Robert - 51 years old and proud of it.",
+  "Kwame here - just hit the big 5-0 this year."
+)
+type_person <- type_object(name = type_string(), age = type_number())
+\dontrun{
+data <- batch_chat_structured(
+  chat = chat,
+  prompts = prompts,
+  path = "people-data.json",
+  type = type_person
+)
+data
+}
+\dontshow{\}) # examplesIf}
+}
diff --git a/man/parallel_chat.Rd b/man/parallel_chat.Rd
index d5428d0ac..9a22b05f7 100644
--- a/man/parallel_chat.Rd
+++ b/man/parallel_chat.Rd
@@ -64,6 +64,10 @@ property.
 If you have multiple prompts, you can submit them in parallel. This is
 typically considerably faster than submitting them in sequence, especially
 with Gemini and OpenAI.
+
+If you're using \code{\link[=chat_openai]{chat_openai()}} or \code{\link[=chat_anthropic]{chat_anthropic()}} and you're willing
+to wait longer, you might want to use \code{\link[=batch_chat]{batch_chat()}} instead, as it comes
+with a 50\% discount in return for taking up to 24 hours.
 }
 \examples{
 \dontshow{if (ellmer::has_credentials("openai")) (if (getRversion() >= "3.4") withAutoprint else force)(\{ # examplesIf}
diff --git a/tests/testthat/_snaps/batch-chat.md b/tests/testthat/_snaps/batch-chat.md
new file mode 100644
index 000000000..6934785af
--- /dev/null
+++ b/tests/testthat/_snaps/batch-chat.md
@@ -0,0 +1,37 @@
+# errors if chat/provider/prompts don't match previous run
+
+    Code
+      batch_chat(chat, prompts, path)
+    Condition
+      Error in `batch_chat()`:
+      ! provider, prompts, and user_turns don't match stored values.
+      i Do you need to pick a different `path`?
+
+# informative error for bad inputs
+
+    Code
+      batch_chat("x")
+    Condition
+      Error in `batch_chat()`:
+      ! `chat` must be a <Chat> object.
+    Code
+      batch_chat(chat_ollama)
+    Condition
+      Error in `batch_chat()`:
+      ! Batch requests are not currently supported by this provider.
+    Code
+      batch_chat(chat_openai, "a")
+    Condition
+      Error in `batch_chat()`:
+      ! `prompts` must be a list or prompt, not the string "a".
+    Code
+      batch_chat(chat_openai, list("a"), path = 1)
+    Condition
+      Error in `batch_chat()`:
+      ! `path` must be a single string, not the number 1.
+    Code
+      batch_chat(chat_openai, list("a"), path = "x", wait = 1)
+    Condition
+      Error in `batch_chat()`:
+      ! `wait` must be `TRUE` or `FALSE`, not the number 1.
+
diff --git a/tests/testthat/batch/state-capitals.json b/tests/testthat/batch/state-capitals.json
new file mode 100644
index 000000000..a6cc1ed66
--- /dev/null
+++ b/tests/testthat/batch/state-capitals.json
@@ -0,0 +1,198 @@
+{
+  "version": 1,
+  "stage": "done",
+  "batch": {
+    "id": "batch_6826575e8994819083aca31bb6d307de",
+    "object": "batch",
+    "endpoint": "/v1/chat/completions",
+    "errors": {},
+    "input_file_id": "file-M1HWkC3RwUDPgaWtZYjjcy",
+    "completion_window": "24h",
+    "status": "completed",
+    "output_file_id": "file-SdRQAoNeNzaFYr3YFb5hcp",
+    "error_file_id": {},
+    "created_at": 1747343198,
+    "in_progress_at": 1747343199,
+    "expires_at": 1747429598,
+    "finalizing_at": 1747343399,
+    "completed_at": 1747343400,
+    "failed_at": {},
+    "expired_at": {},
+    "cancelling_at": {},
+    "cancelled_at": {},
+    "request_counts": {
+      "total": 4,
+      "completed": 4,
+      "failed": 0
+    },
+    "metadata": {}
+  },
+  "results": [
+    {
+      "status_code": 200,
+      "request_id": "6c41082f8debff173736a0b52e07d948",
+      "body": {
+        "id": "chatcmpl-BXa7llR1CJFH3KbFBLJf4ZvvYEUj4",
+        "object": "chat.completion",
+        "created": 1747343385,
+        "model": "gpt-4.1-nano-2025-04-14",
+        "choices": [
+          {
+            "index": 0,
+            "message": {
+              "role": "assistant",
+              "content": "Des Moines",
+              "refusal": {},
+              "annotations": []
+            },
+            "logprobs": {},
+            "finish_reason": "stop"
+          }
+        ],
+        "usage": {
+          "prompt_tokens": 20,
+          "completion_tokens": 2,
+          "total_tokens": 22,
+          "prompt_tokens_details": {
+            "cached_tokens": 0,
+            "audio_tokens": 0
+          },
+          "completion_tokens_details": {
+            "reasoning_tokens": 0,
+            "audio_tokens": 0,
+            "accepted_prediction_tokens": 0,
+            "rejected_prediction_tokens": 0
+          }
+        },
+        "service_tier": "default",
+        "system_fingerprint": "fp_eede8f0d45"
+      }
+    },
+    {
+      "status_code": 200,
+      "request_id": "881526127f188f503beb9e98d7933d01",
+      "body": {
+        "id": "chatcmpl-BXa6CU6GypSA9ODF6TKhpkHPP8jlI",
+        "object": "chat.completion",
+        "created": 1747343288,
+        "model": "gpt-4.1-nano-2025-04-14",
+        "choices": [
+          {
+            "index": 0,
+            "message": {
+              "role": "assistant",
+              "content": "Albany",
+              "refusal": {},
+              "annotations": []
+            },
+            "logprobs": {},
+            "finish_reason": "stop"
+          }
+        ],
+        "usage": {
+          "prompt_tokens": 21,
+          "completion_tokens": 2,
+          "total_tokens": 23,
+          "prompt_tokens_details": {
+            "cached_tokens": 0,
+            "audio_tokens": 0
+          },
+          "completion_tokens_details": {
+            "reasoning_tokens": 0,
+            "audio_tokens": 0,
+            "accepted_prediction_tokens": 0,
+            "rejected_prediction_tokens": 0
+          }
+        },
+        "service_tier": "default",
+        "system_fingerprint": "fp_eede8f0d45"
+      }
+    },
+    {
+      "status_code": 200,
+      "request_id": "d35aa9cb5535dd1ea70e1c5e3c469791",
+      "body": {
+        "id": "chatcmpl-BXa6316yijIIu48fQeazyXljyjTXk",
+        "object": "chat.completion",
+        "created": 1747343279,
+        "model": "gpt-4.1-nano-2025-04-14",
+        "choices": [
+          {
+            "index": 0,
+            "message": {
+              "role": "assistant",
+              "content": "Sacramento",
+              "refusal": {},
+              "annotations": []
+            },
+            "logprobs": {},
+            "finish_reason": "stop"
+          }
+        ],
+        "usage": {
+          "prompt_tokens": 20,
+          "completion_tokens": 3,
+          "total_tokens": 23,
+          "prompt_tokens_details": {
+            "cached_tokens": 0,
+            "audio_tokens": 0
+          },
+          "completion_tokens_details": {
+            "reasoning_tokens": 0,
+            "audio_tokens": 0,
+            "accepted_prediction_tokens": 0,
+            "rejected_prediction_tokens": 0
+          }
+        },
+        "service_tier": "default",
+        "system_fingerprint": "fp_eede8f0d45"
+      }
+    },
+    {
+      "status_code": 200,
+      "request_id": "fb17b486b4053f56120614c6fbffe571",
+      "body": {
+        "id": "chatcmpl-BXa6aPBtnciow6qc6gjEPMJ7DjgY7",
+        "object": "chat.completion",
+        "created": 1747343312,
+        "model": "gpt-4.1-nano-2025-04-14",
+        "choices": [
+          {
+            "index": 0,
+            "message": {
+              "role": "assistant",
+              "content": "Austin",
+              "refusal": {},
+              "annotations": []
+            },
+            "logprobs": {},
+            "finish_reason": "stop"
+          }
+        ],
+        "usage": {
+          "prompt_tokens": 20,
+          "completion_tokens": 2,
+          "total_tokens": 22,
+          "prompt_tokens_details": {
+            "cached_tokens": 0,
+            "audio_tokens": 0
+          },
+          "completion_tokens_details": {
+            "reasoning_tokens": 0,
+            "audio_tokens": 0,
+            "accepted_prediction_tokens": 0,
+            "rejected_prediction_tokens": 0
+          }
+        },
+        "service_tier": "default",
+        "system_fingerprint": "fp_eede8f0d45"
+      }
+    }
+  ],
+  "started_at": 1747343197,
+  "hash": {
+    "provider": "de1de607a44b8964f051074a91a3787e",
+    "prompts": "b8eafe281e3cc5113058d9722be3e295",
+    "user_turns": "8c1302088b9bc30258d1191db0d35705"
+  }
+}
diff --git a/tests/testthat/batch/state-name.json b/tests/testthat/batch/state-name.json
new file mode 100644
index 000000000..8d0d434c9
--- /dev/null
+++ b/tests/testthat/batch/state-name.json
@@ -0,0 +1,198 @@
+{
+  "version": 1,
+  "stage": "done",
+  "batch": {
+    "id": "batch_6826582985488190a4489a615c37539f",
+    "object": "batch",
+    "endpoint": "/v1/chat/completions",
+    "errors": {},
+    "input_file_id": "file-2ohRDLpVLSGcpcpbmMCf79",
+    "completion_window": "24h",
+    "status": "completed",
+    "output_file_id": "file-4kMvA4FaKqXvw59xYMfUJW",
+    "error_file_id": {},
+    "created_at": 1747343401,
+    "in_progress_at": 1747343402,
+    "expires_at": 1747429801,
+    "finalizing_at": 1747343603,
+    "completed_at": 1747343604,
+    "failed_at": {},
+    "expired_at": {},
+    "cancelling_at": {},
+    "cancelled_at": {},
+    "request_counts": {
+      "total": 4,
+      "completed": 4,
+      "failed": 0
+    },
+    "metadata": {}
+  },
+  "results": [
+    {
+      "status_code": 200,
+      "request_id": "e308e441d289045e722626dd080ef6b0",
+      "body": {
+        "id": "chatcmpl-BXaAzMTurodvGwYv20MMNPkn9Ngaz",
+        "object": "chat.completion",
+        "created": 1747343585,
+        "model": "gpt-4.1-nano-2025-04-14",
+        "choices": [
+          {
+            "index": 0,
+            "message": {
+              "role": "assistant",
+              "content": "{\"wrapper\":{\"name\":\"Des Moines\"}}",
+              "refusal": {},
+              "annotations": []
+            },
+            "logprobs": {},
+            "finish_reason": "stop"
+          }
+        ],
+        "usage": {
+          "prompt_tokens": 64,
+          "completion_tokens": 8,
+          "total_tokens": 72,
+          "prompt_tokens_details": {
+            "cached_tokens": 0,
+            "audio_tokens": 0
+          },
+          "completion_tokens_details": {
+            "reasoning_tokens": 0,
+            "audio_tokens": 0,
+            "accepted_prediction_tokens": 0,
+            "rejected_prediction_tokens": 0
+          }
+        },
+        "service_tier": "default",
+        "system_fingerprint": "fp_eede8f0d45"
+      }
+    },
+    {
+      "status_code": 200,
+      "request_id": "4ec4485816428a52a968d6f645a4d9b8",
+      "body": {
+        "id": "chatcmpl-BXaAz0hjcGCfGBDTKvDnQNEnTfJiw",
+        "object": "chat.completion",
+        "created": 1747343585,
+        "model": "gpt-4.1-nano-2025-04-14",
+        "choices": [
+          {
+            "index": 0,
+            "message": {
+              "role": "assistant",
+              "content": "{\"wrapper\":{\"name\":\"Albany\"}}",
+              "refusal": {},
+              "annotations": []
+            },
+            "logprobs": {},
+            "finish_reason": "stop"
+          }
+        ],
+        "usage": {
+          "prompt_tokens": 65,
+          "completion_tokens": 8,
+          "total_tokens": 73,
+          "prompt_tokens_details": {
+            "cached_tokens": 0,
+            "audio_tokens": 0
+          },
+          "completion_tokens_details": {
+            "reasoning_tokens": 0,
+            "audio_tokens": 0,
+            "accepted_prediction_tokens": 0,
+            "rejected_prediction_tokens": 0
+          }
+        },
+        "service_tier": "default",
+        "system_fingerprint": "fp_eede8f0d45"
+      }
+    },
+    {
+      "status_code": 200,
+      "request_id": "cb7815c6867bb534580831eaedacc249",
+      "body": {
+        "id": "chatcmpl-BXaAz7e7k7zRVVcrxrHp1YUpI6eXw",
+        "object": "chat.completion",
+        "created": 1747343585,
+        "model": "gpt-4.1-nano-2025-04-14",
+        "choices": [
+          {
+            "index": 0,
+            "message": {
+              "role": "assistant",
+              "content": "{\"wrapper\":{\"name\":\"Sacramento\"}}",
+              "refusal": {},
+              "annotations": []
+            },
+            "logprobs": {},
+            "finish_reason": "stop"
+          }
+        ],
+        "usage": {
+          "prompt_tokens": 64,
+          "completion_tokens": 8,
+          "total_tokens": 72,
+          "prompt_tokens_details": {
+            "cached_tokens": 0,
+            "audio_tokens": 0
+          },
+          "completion_tokens_details": {
+            "reasoning_tokens": 0,
+            "audio_tokens": 0,
+            "accepted_prediction_tokens": 0,
+            "rejected_prediction_tokens": 0
+          }
+        },
+        "service_tier": "default",
+        "system_fingerprint": "fp_eede8f0d45"
+      }
+    },
+    {
+      "status_code": 200,
+      "request_id": "2844ea0eaf101fd5a9c684d070036749",
+      "body": {
+        "id": "chatcmpl-BXa839Kt3smWySomfLXaaWbVnLAWm",
+        "object": "chat.completion",
+        "created": 1747343403,
+        "model": "gpt-4.1-nano-2025-04-14",
+        "choices": [
+          {
+            "index": 0,
+            "message": {
+              "role": "assistant",
+              "content": "{\"wrapper\":{\"name\":\"Austin\"}}",
+              "refusal": {},
+              "annotations": []
+            },
+            "logprobs": {},
+            "finish_reason": "stop"
+          }
+        ],
+        "usage": {
+          "prompt_tokens": 64,
+          "completion_tokens": 7,
+          "total_tokens": 71,
+          "prompt_tokens_details": {
+            "cached_tokens": 0,
+            "audio_tokens": 0
+          },
+          "completion_tokens_details": {
+            "reasoning_tokens": 0,
+            "audio_tokens": 0,
+            "accepted_prediction_tokens": 0,
+            "rejected_prediction_tokens": 0
+          }
+        },
+        "service_tier": "default",
+        "system_fingerprint": "fp_eede8f0d45"
+      }
+    }
+  ],
+  "started_at": 1747343400,
+  "hash": {
+    "provider": "de1de607a44b8964f051074a91a3787e",
+    "prompts": "b8eafe281e3cc5113058d9722be3e295",
+    "user_turns": "8c1302088b9bc30258d1191db0d35705"
+  }
+}
diff --git a/tests/testthat/test-batch-chat.R b/tests/testthat/test-batch-chat.R
new file mode 100644
index 000000000..967507d3c
--- /dev/null
+++ b/tests/testthat/test-batch-chat.R
@@ -0,0 +1,125 @@
+test_that("can get chats/data from completed request", {
+  chat <- chat_openai_test()
+
+  prompts <- list(
+    "What's the capital of Iowa?",
+    "What's the capital of New York?",
+    "What's the capital of California?",
+    "What's the capital of Texas?"
+  )
+  chats <- batch_chat(
+    chat,
+    prompts,
+    path = test_path("batch/state-capitals.json")
+  )
+  expect_length(chats, 4)
+
+  type_state <- type_object(name = type_string("State name"))
+  data <- batch_chat_structured(
+    chat,
+    prompts,
+    path = test_path("batch/state-name.json"),
+    type = type_state
+  )
+  expect_equal(nrow(data), 4)
+})
+
+test_that("errors if chat/provider/prompts don't match previous run", {
+  chat <- chat_anthropic_test(system_prompt = "Be cool")
+  prompts <- list("What's the capital of Iowa?")
+  path <- test_path("batch/state-capitals.json")
+  expect_snapshot(batch_chat(chat, prompts, path), error = TRUE)
+})
+
+test_that("steps through in logical order, writing to disk at end step", {
+  chat <- chat_openai_test()
+  prompts <- list("What's your name")
+  local_mocked_bindings(
+    batch_submit = function(...) list(id = "123"),
+    batch_poll = function(...) list(id = "123", results = TRUE),
+    batch_status = function(...) list(working = FALSE),
+    batch_retrieve = function(...) list(x = 1, y = 2)
+  )
+
+  path <- withr::local_tempfile()
+  read_stage <- function() jsonlite::read_json(path)$stage
+
+  job <- BatchJob$new(chat, prompts, path)
+  completed <- \() batch_chat_completed(chat, prompts, path)
+
+  expect_equal(job$stage, "submitting")
+  expect_false(completed())
+
+  job$step()
+  expect_equal(job$stage, "waiting")
+  expect_equal(read_stage(), "waiting")
+  expect_equal(job$batch, list(id = "123"))
+  expect_true(completed())
+
+  job$step()
+  expect_equal(job$stage, "retrieving")
+  expect_equal(read_stage(), "retrieving")
+  expect_equal(job$batch, list(id = "123", results = TRUE))
+  expect_true(completed())
+
+  job$step()
+  expect_equal(job$stage, "done")
+  expect_equal(read_stage(), "done")
+  expect_equal(job$results, list(x = 1, y = 2))
+  expect_true(completed())
+})
+
+test_that("can run all steps at once", {
+  local_mocked_bindings(
+    batch_submit = function(...) list(id = "123"),
+    batch_poll = function(...) list(id = "123", results = TRUE),
+    batch_status = function(...) list(working = FALSE),
+    batch_retrieve = function(...) list(x = 1, y = 2)
+  )
+
+  path <- withr::local_tempfile()
+  job <- BatchJob$new(
+    chat = chat_openai_test(),
+    prompts = list("What's your name"),
+    path = path
+  )
+  job$step_until_done()
+  expect_equal(job$stage, "done")
+  expect_equal(job$results, list(x = 1, y = 2))
+})
+
+test_that("errors if wait = FALSE and not complete", {
+  local_mocked_bindings(
+    batch_submit = function(...) list(id = "123"),
+    batch_poll = function(...) list(id = "123", results = TRUE),
+    batch_status = function(...) list(working = TRUE)
+  )
+
+  path <- withr::local_tempfile()
+  job <- BatchJob$new(
+    chat = chat_openai_test(),
+    prompts = list("What's your name"),
+    path = path,
+    wait = FALSE
+  )
+  expect_equal(job$step_until_done(), NULL)
+})
+
+test_that("informative error for bad inputs", {
+  chat_openai <- chat_openai_test()
+  chat_ollama <- chat_openai_test()
+  chat_ollama$.__enclos_env__$private$provider <- ProviderOllama(
+    "ollama",
+    "model",
+    "base_url",
+    api_key = "api_key"
+  )
+
+  expect_snapshot(error = TRUE, {
+    batch_chat("x")
+    batch_chat(chat_ollama)
+    batch_chat(chat_openai, "a")
+    batch_chat(chat_openai, list("a"), path = 1)
+    batch_chat(chat_openai, list("a"), path = "x", wait = 1)
+  })
+})


Reviews:
user: login: gadenbuie
body: > In particular, what do you think of the required `path` argument, and is it correct to throw an error if `wait = FALSE`.

I like the API! But I think it'd be helpful to lean into the state JSON storage a bit more and break out `batch_chat_status()` and `batch_chat_results()` into user-facing functions.

After the initial `batch_chat(chat, prompts, "state.json")` call, you could store the provider in the `state.json` so that you can check the status or the results without having to pass in `chat` or `prompts`:

```r
chat <- chat_openai(model = "gpt-4.1-nano)
prompts <- interpolate("What do people from {{state.name}} bring to a potluck dinner?")
batch_chat(chat, prompts, path = "states.json")

# Walk away, restart your R session, etc.
batch_chat_status("states.json")
#> "waiting"

# later...
batch_chat_status("states.json")
#> "ready"

chats <- batch_chat_results("states.json")
chats[[which(state.abb == "GA")]]
#> <Chat OpenAI/gpt-4.1-nano turns=2 tokens=19/243 $0.00>
#> ── user [19] ─────────────────────────────────────────
#> What do people from Georgia bring to a potluck dinner?
#> ── assistant [243] ───────────────────────────────────
#> People from Georgia often bring a variety of delicious
#> dishes to a potluck dinner that reflect the state's 
#> rich culinary traditions. Some popular options include:
#> 
#> 1. **Fried Chicken** – Crispy and flavorful, a Southern staple.
#> 2. **Grits Casserole** – Creamy or cheesy grits baked to perfection.
#> 3. **Pecan Pie or Peach Cobbler** – Classic Southern desserts featuring Georgia’s famous pecans and peaches.
```

With that API, I might suggest making `path` optional in `batch_chat()`, using something like `ellmer-batch-{hash(list(chat, prompts))}.json`. When `wait = TRUE`, you wouldn't necessarily tell the user about the file (and you might even clean it up when exiting `batch_chat()`), but you could catch interrupts and print the file name if they exit `batch_chat()` early.

When `wait = FALSE`, `batch_chat()` could return the `path` invisibly, possibly with a message if `path` was automatically created, e.g. something like

```r
batch_chat(chat, prompts, wait = FALSE)
#> Started batch chat with 50 prompts.
#> i Call `batch_chat_status("ellmer-batch-dabc83e9be961b3ffa768d0d2fb8ac10.json") to check the status.
```

And maybe also `batch_chat_status()` could have a `wait` argument that's `FALSE` by default, but if `TRUE` would start up the progress bar and block until complete.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T13:12:03Z
user: login: gadenbuie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T13:17:35Z
user: login: hadley
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-15T13:24:16Z
user: login: gadenbuie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T13:05:40Z
user: login: hadley
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T13:30:34Z
user: login: gadenbuie
body: Looks great! I really like the feature and the internal API with R6 class is quite clean. No big comments, all just about final polish
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-16T13:46:51Z
user: login: hadley
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T14:11:31Z
user: login: hadley
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T14:13:41Z
user: login: hadley
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T14:16:31Z
user: login: hadley
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T14:18:07Z
user: login: hadley
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T14:18:47Z

Review comments:
diff_hunk: @@ -0,0 +1,129 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` is currently implemented for OpenAI and Anthropic, where it
+#' uses their batch API which allow you to submit multiple requests
+#' simultaenously. You get the results back within 24 hours in return for paying
path: R/batch-chat.R
user: login: gadenbuie
body: ```suggestion
#' simultaneously. You get the results back within 24 hours in return for paying
```
created_at: 2025-05-15T12:25:34Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,129 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` is currently implemented for OpenAI and Anthropic, where it
+#' uses their batch API which allow you to submit multiple requests
+#' simultaenously. You get the results back within 24 hours in return for paying
+#' 50% less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, you
+#' may want to use [parallel_chat()] instead.
+#'
+#' Since batched requests can take a long time to complete, `batch_chat()`
+#' requires a file path that is used to store information about the batch so
+#' you never lose any work. You can either set `wait = FALSE` or simply
+#' interrupt the waiting process, then later, call `batch_chat()` to resume
+#' where you left off. Once you're done with the chat results, delete the
+#' file you created to avoid re-using previous results.
+#'
+#' This API is marked as experimental, since I don't know how to helpfully
+#' deal with errors. Fortunately they don't seem to be common, but if you
+#' have ideas, please let me know!
+#'
+#' @inheritParams parallel_chat
+#' @param path Path to file (with `.json` extension) to store state.
+#' @param wait If `TRUE`, will wait for batch to complete. If `FALSE`,
+#'   it will check once and error if the job is not complete.
+#' @examplesIf has_credentials("claude")
+#' chat <- chat_openai()
+#'
+#' # Chat ----------------------------------------------------------------------
+#' country <- c("Canada", "New Zealand", "Jamaica", "United States")
+#' prompts <- interpolate("What's the capital of {{country}}?")
+#' \dontrun{
+#' chats <- batch_chat(chat, prompts, path = "capitals.json")
+#' chats
+#' }
+batch_chat <- function(chat, prompts, path, wait = TRUE) {
+  check_chat(chat)
+  provider <- chat$get_provider()
+  check_has_batch_support(provider)
+
+  user_turns <- as_user_turns(prompts)
+
+  if (file.exists(path)) {
+    state <- jsonlite::read_json(path)
+  } else {
+    state <- list(stage = "submitting", batch = NULL, results = NULL)
+  }
+  save_state <- function() {
+    jsonlite::write_json(
+      state,
+      path,
+      auto_unbox = TRUE,
+      pretty = TRUE
+    )
+  }
+
+  if (state$stage == "submitting") {
+    existing <- chat$get_turns(include_system_prompt = TRUE)
+    conversations <- append_turns(list(existing), user_turns)
+    state$batch <- batch_submit(provider, conversations)
+    state$stage <- "waiting"
+    save_state()
+  }
+
+  if (state$stage == "waiting") {
+    # want to ensure that we always cycle once, even when wait = FALSE
+    state$batch <- batch_poll(provider, state$batch)
+    status <- batch_status(provider, state$batch)
+    save_state()
+
+    if (status$working && wait) {
+      cli::cli_progress_bar(
+        format = paste(
+          "{cli::pb_spin} Processing... ",
+          "{status$n_processing} -> {cli::col_green({status$n_succeeded})} / {cli::col_red({status$n_failed})} ",
+          "[{cli::pb_elapsed}]"
path: R/batch-chat.R
user: login: gadenbuie
body: It'd be nice if the timer shows the duration of the batch using `batch.created_at` (especially if you're coming back to check on a batch).
created_at: 2025-05-15T12:38:51Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,129 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` is currently implemented for OpenAI and Anthropic, where it
+#' uses their batch API which allow you to submit multiple requests
+#' simultaenously. You get the results back within 24 hours in return for paying
+#' 50% less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, you
+#' may want to use [parallel_chat()] instead.
+#'
+#' Since batched requests can take a long time to complete, `batch_chat()`
+#' requires a file path that is used to store information about the batch so
+#' you never lose any work. You can either set `wait = FALSE` or simply
+#' interrupt the waiting process, then later, call `batch_chat()` to resume
+#' where you left off. Once you're done with the chat results, delete the
+#' file you created to avoid re-using previous results.
+#'
+#' This API is marked as experimental, since I don't know how to helpfully
+#' deal with errors. Fortunately they don't seem to be common, but if you
+#' have ideas, please let me know!
+#'
+#' @inheritParams parallel_chat
+#' @param path Path to file (with `.json` extension) to store state.
+#' @param wait If `TRUE`, will wait for batch to complete. If `FALSE`,
+#'   it will check once and error if the job is not complete.
+#' @examplesIf has_credentials("claude")
+#' chat <- chat_openai()
+#'
+#' # Chat ----------------------------------------------------------------------
+#' country <- c("Canada", "New Zealand", "Jamaica", "United States")
+#' prompts <- interpolate("What's the capital of {{country}}?")
+#' \dontrun{
+#' chats <- batch_chat(chat, prompts, path = "capitals.json")
+#' chats
+#' }
+batch_chat <- function(chat, prompts, path, wait = TRUE) {
+  check_chat(chat)
+  provider <- chat$get_provider()
+  check_has_batch_support(provider)
+
+  user_turns <- as_user_turns(prompts)
+
+  if (file.exists(path)) {
+    state <- jsonlite::read_json(path)
+  } else {
+    state <- list(stage = "submitting", batch = NULL, results = NULL)
+  }
+  save_state <- function() {
+    jsonlite::write_json(
+      state,
+      path,
+      auto_unbox = TRUE,
+      pretty = TRUE
+    )
+  }
+
+  if (state$stage == "submitting") {
+    existing <- chat$get_turns(include_system_prompt = TRUE)
+    conversations <- append_turns(list(existing), user_turns)
+    state$batch <- batch_submit(provider, conversations)
+    state$stage <- "waiting"
+    save_state()
+  }
+
+  if (state$stage == "waiting") {
+    # want to ensure that we always cycle once, even when wait = FALSE
+    state$batch <- batch_poll(provider, state$batch)
+    status <- batch_status(provider, state$batch)
+    save_state()
+
+    if (status$working && wait) {
+      cli::cli_progress_bar(
+        format = paste(
+          "{cli::pb_spin} Processing... ",
+          "{status$n_processing} -> {cli::col_green({status$n_succeeded})} / {cli::col_red({status$n_failed})} ",
+          "[{cli::pb_elapsed}]"
path: R/batch-chat.R
user: login: gadenbuie
body: Another minor comment: I'd prefer using words in addition to colors to communicate status here and I think there's room for it.

```
# Instead of
⠙ Processing...  46 -> 4 / 0 [5.6s]

# Something like this
⠙ Processing...  46 pending | 4 done | 0 failed [0:05]

# Or maybe
⠙ Processing... [0:05] 46 pending | 4 done | 0 failed
```
created_at: 2025-05-15T13:17:35Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,129 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` is currently implemented for OpenAI and Anthropic, where it
+#' uses their batch API which allow you to submit multiple requests
+#' simultaenously. You get the results back within 24 hours in return for paying
+#' 50% less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, you
+#' may want to use [parallel_chat()] instead.
+#'
+#' Since batched requests can take a long time to complete, `batch_chat()`
+#' requires a file path that is used to store information about the batch so
+#' you never lose any work. You can either set `wait = FALSE` or simply
+#' interrupt the waiting process, then later, call `batch_chat()` to resume
+#' where you left off. Once you're done with the chat results, delete the
+#' file you created to avoid re-using previous results.
+#'
+#' This API is marked as experimental, since I don't know how to helpfully
+#' deal with errors. Fortunately they don't seem to be common, but if you
+#' have ideas, please let me know!
+#'
+#' @inheritParams parallel_chat
+#' @param path Path to file (with `.json` extension) to store state.
+#' @param wait If `TRUE`, will wait for batch to complete. If `FALSE`,
+#'   it will check once and error if the job is not complete.
+#' @examplesIf has_credentials("claude")
+#' chat <- chat_openai()
+#'
+#' # Chat ----------------------------------------------------------------------
+#' country <- c("Canada", "New Zealand", "Jamaica", "United States")
+#' prompts <- interpolate("What's the capital of {{country}}?")
+#' \dontrun{
+#' chats <- batch_chat(chat, prompts, path = "capitals.json")
+#' chats
+#' }
+batch_chat <- function(chat, prompts, path, wait = TRUE) {
+  check_chat(chat)
+  provider <- chat$get_provider()
+  check_has_batch_support(provider)
+
+  user_turns <- as_user_turns(prompts)
+
+  if (file.exists(path)) {
+    state <- jsonlite::read_json(path)
+  } else {
+    state <- list(stage = "submitting", batch = NULL, results = NULL)
+  }
+  save_state <- function() {
+    jsonlite::write_json(
+      state,
+      path,
+      auto_unbox = TRUE,
+      pretty = TRUE
+    )
+  }
+
+  if (state$stage == "submitting") {
+    existing <- chat$get_turns(include_system_prompt = TRUE)
+    conversations <- append_turns(list(existing), user_turns)
+    state$batch <- batch_submit(provider, conversations)
+    state$stage <- "waiting"
+    save_state()
+  }
+
+  if (state$stage == "waiting") {
+    # want to ensure that we always cycle once, even when wait = FALSE
+    state$batch <- batch_poll(provider, state$batch)
+    status <- batch_status(provider, state$batch)
+    save_state()
+
+    if (status$working && wait) {
+      cli::cli_progress_bar(
+        format = paste(
+          "{cli::pb_spin} Processing... ",
+          "{status$n_processing} -> {cli::col_green({status$n_succeeded})} / {cli::col_red({status$n_failed})} ",
+          "[{cli::pb_elapsed}]"
path: R/batch-chat.R
user: login: hadley
body: Oh yeah, good idea. Let me figure out how to do that.
created_at: 2025-05-15T13:24:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,333 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` and `batch_chat_structured()` currently only work with
+#' [chat_openai()] and [chat_anthropic()]. They use the OpenAI and Anthropic
+#' batch APIs which allow you to submit multiple requests simultaneously.
+#' The results can take up to 24 hours to complete, but in return you pay 50%
+#' less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, or
+#' you're working with a different provider, you may want to use
+#' [parallel_chat()] instead.
+#'
+#' Since batched requests can take a long time to complete, `batch_chat()`
+#' requires a file path that is used to store information about the batch so
+#' you never lose any work. You can either set `wait = FALSE` or simply
+#' interrupt the waiting process, then later, either call `batch_chat()` to
+#' resume where you left off or call `batch_chat_completed()` to see if the
+#' results are ready to retrieve. Once you have the results, you can delete
+#' the file.
+#'
+#' This API is marked as experimental since I don't yet know how to handle
+#' errors in the most helpful way. Fortunately they don't seem to be common,
+#' but if you have ideas, please let me know!
+#'
+#' @inheritParams parallel_chat
+#' @param path Path to file (with `.json` extension) to store state.
+#'
+#'   The file records a hash of the provider, the prompts, and the existing
+#'   chat turns. If you attempt to reuse the same file with any of these being
+#'   different, you'll get an error.
+#' @param wait If `TRUE`, will wait for batch to complete. If `FALSE`,
+#'   it will return `NULL` if the batch is not complete, and you can retrieve
+#'   the results later by re-running `batch_chat()` when
+#'   `batch_chat_completed()` is `TRUE`.
+#' @examplesIf has_credentials("openai")
+#' chat <- chat_openai(model = "gpt-4.1-nano")
+#'
+#' # Chat ----------------------------------------------------------------------
+#'
+#' prompts <- interpolate("What do people from {{state.name}} bring to a potluck dinner?")
+#' \dontrun{
+#' chats <- batch_chat(chat, prompts, path = "potluck.json")
+#' chats
+#' }
+#'
+#' # Structured data -----------------------------------------------------------
+#' prompts <- list(
+#'   "I go by Alex. 42 years on this planet and counting.",
+#'   "Pleased to meet you! I'm Jamal, age 27.",
+#'   "They call me Li Wei. Nineteen years young.",
+#'   "Fatima here. Just celebrated my 35th birthday last week.",
+#'   "The name's Robert - 51 years old and proud of it.",
+#'   "Kwame here - just hit the big 5-0 this year."
+#' )
+#' type_person <- type_object(name = type_string(), age = type_number())
+#' \dontrun{
+#' data <- batch_chat_structured(
+#'   chat = chat,
+#'   prompts = prompts,
+#'   path = "people-data.json",
+#'   type = type_person
+#' )
+#' data
+#' }
+#'
+batch_chat <- function(chat, prompts, path, wait = TRUE) {
path: R/batch-chat.R
user: login: gadenbuie
body: ```suggestion
#'
#' @export
batch_chat <- function(chat, prompts, path, wait = TRUE) {
```
created_at: 2025-05-16T13:05:40Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,333 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` and `batch_chat_structured()` currently only work with
+#' [chat_openai()] and [chat_anthropic()]. They use the OpenAI and Anthropic
+#' batch APIs which allow you to submit multiple requests simultaneously.
+#' The results can take up to 24 hours to complete, but in return you pay 50%
+#' less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, or
path: R/batch-chat.R
user: login: gadenbuie
body: This sentence reminded me that I think it'd be useful to include links to OpenAI and Anthropic's batched message docs, since these will have up-to-date information on completion time, pricing and how long results are stored.
created_at: 2025-05-16T13:07:56Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,333 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` and `batch_chat_structured()` currently only work with
+#' [chat_openai()] and [chat_anthropic()]. They use the OpenAI and Anthropic
+#' batch APIs which allow you to submit multiple requests simultaneously.
+#' The results can take up to 24 hours to complete, but in return you pay 50%
+#' less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, or
+#' you're working with a different provider, you may want to use
+#' [parallel_chat()] instead.
+#'
+#' Since batched requests can take a long time to complete, `batch_chat()`
+#' requires a file path that is used to store information about the batch so
+#' you never lose any work. You can either set `wait = FALSE` or simply
+#' interrupt the waiting process, then later, either call `batch_chat()` to
+#' resume where you left off or call `batch_chat_completed()` to see if the
+#' results are ready to retrieve. Once you have the results, you can delete
+#' the file.
path: R/batch-chat.R
user: login: gadenbuie
body: Maybe mention that you could also keep the file to cache the results?

And maybe also that `batch_chat()` stores the results in the file (rather than retrieving them anew from the provider) so as long as you have the file you can access the results. Re-running a script and using cached `batch_chat()` results each time as Tomasz mentioned sounds both like a feature and a footgun.
created_at: 2025-05-16T13:15:02Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,333 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` and `batch_chat_structured()` currently only work with
+#' [chat_openai()] and [chat_anthropic()]. They use the OpenAI and Anthropic
+#' batch APIs which allow you to submit multiple requests simultaneously.
+#' The results can take up to 24 hours to complete, but in return you pay 50%
+#' less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, or
+#' you're working with a different provider, you may want to use
+#' [parallel_chat()] instead.
+#'
+#' Since batched requests can take a long time to complete, `batch_chat()`
+#' requires a file path that is used to store information about the batch so
+#' you never lose any work. You can either set `wait = FALSE` or simply
+#' interrupt the waiting process, then later, either call `batch_chat()` to
+#' resume where you left off or call `batch_chat_completed()` to see if the
+#' results are ready to retrieve. Once you have the results, you can delete
+#' the file.
+#'
+#' This API is marked as experimental since I don't yet know how to handle
+#' errors in the most helpful way. Fortunately they don't seem to be common,
+#' but if you have ideas, please let me know!
+#'
+#' @inheritParams parallel_chat
+#' @param path Path to file (with `.json` extension) to store state.
+#'
+#'   The file records a hash of the provider, the prompts, and the existing
+#'   chat turns. If you attempt to reuse the same file with any of these being
+#'   different, you'll get an error.
+#' @param wait If `TRUE`, will wait for batch to complete. If `FALSE`,
+#'   it will return `NULL` if the batch is not complete, and you can retrieve
+#'   the results later by re-running `batch_chat()` when
+#'   `batch_chat_completed()` is `TRUE`.
+#' @examplesIf has_credentials("openai")
+#' chat <- chat_openai(model = "gpt-4.1-nano")
+#'
+#' # Chat ----------------------------------------------------------------------
+#'
+#' prompts <- interpolate("What do people from {{state.name}} bring to a potluck dinner?")
+#' \dontrun{
+#' chats <- batch_chat(chat, prompts, path = "potluck.json")
+#' chats
+#' }
+#'
+#' # Structured data -----------------------------------------------------------
+#' prompts <- list(
+#'   "I go by Alex. 42 years on this planet and counting.",
+#'   "Pleased to meet you! I'm Jamal, age 27.",
+#'   "They call me Li Wei. Nineteen years young.",
+#'   "Fatima here. Just celebrated my 35th birthday last week.",
+#'   "The name's Robert - 51 years old and proud of it.",
+#'   "Kwame here - just hit the big 5-0 this year."
+#' )
+#' type_person <- type_object(name = type_string(), age = type_number())
+#' \dontrun{
+#' data <- batch_chat_structured(
+#'   chat = chat,
+#'   prompts = prompts,
+#'   path = "people-data.json",
+#'   type = type_person
+#' )
+#' data
+#' }
+#'
+batch_chat <- function(chat, prompts, path, wait = TRUE) {
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    path = path,
+    wait = wait
+  )
+  job$step_until_done()
+
+  assistant_turns <- job$result_turns()
+  map2(job$user_turns, assistant_turns, function(user, assistant) {
+    if (!is.null(assistant)) {
+      chat$clone()$add_turn(user, assistant)
+    } else {
+      NULL
+    }
+  })
+}
+
+#' @export
+#' @rdname batch_chat
+#' @inheritParams parallel_chat_structured
+batch_chat_structured <- function(
+  chat,
+  prompts,
+  path,
+  type,
+  wait = TRUE,
+  convert = TRUE,
+  include_tokens = FALSE,
+  include_cost = FALSE
+) {
+  check_chat(chat)
+  provider <- chat$get_provider()
+  needs_wrapper <- S7_inherits(provider, ProviderOpenAI)
+
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    type = wrap_type_if_needed(type, needs_wrapper),
+    path = path,
+    wait = wait,
+    call = error_call
+  )
+  job$step_until_done()
+  turns <- job$result_turns()
+
+  multi_convert(
+    provider,
+    turns,
+    type,
+    convert = convert,
+    include_tokens = include_tokens,
+    include_cost = include_cost
+  )
+}
+
+#' @export
+#' @rdname batch_chat
+batch_chat_completed <- function(chat, prompts, path) {
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    path = path
+  )
+  switch(
+    job$stage,
+    "submitting" = FALSE,
+    "waiting" = !job$poll()$working,
+    "retrieving" = TRUE,
+    "done" = TRUE,
+    cli::cli_abort("Unexpected stage: {job$stage}", .internal = TRUE)
+  )
+}
+
+BatchJob <- R6::R6Class(
path: R/batch-chat.R
user: login: gadenbuie
body: Really like the R6 approach!
created_at: 2025-05-16T13:16:58Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,333 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` and `batch_chat_structured()` currently only work with
+#' [chat_openai()] and [chat_anthropic()]. They use the OpenAI and Anthropic
+#' batch APIs which allow you to submit multiple requests simultaneously.
+#' The results can take up to 24 hours to complete, but in return you pay 50%
+#' less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, or
+#' you're working with a different provider, you may want to use
+#' [parallel_chat()] instead.
+#'
+#' Since batched requests can take a long time to complete, `batch_chat()`
+#' requires a file path that is used to store information about the batch so
+#' you never lose any work. You can either set `wait = FALSE` or simply
+#' interrupt the waiting process, then later, either call `batch_chat()` to
+#' resume where you left off or call `batch_chat_completed()` to see if the
+#' results are ready to retrieve. Once you have the results, you can delete
+#' the file.
+#'
+#' This API is marked as experimental since I don't yet know how to handle
+#' errors in the most helpful way. Fortunately they don't seem to be common,
+#' but if you have ideas, please let me know!
+#'
+#' @inheritParams parallel_chat
+#' @param path Path to file (with `.json` extension) to store state.
+#'
+#'   The file records a hash of the provider, the prompts, and the existing
+#'   chat turns. If you attempt to reuse the same file with any of these being
+#'   different, you'll get an error.
+#' @param wait If `TRUE`, will wait for batch to complete. If `FALSE`,
+#'   it will return `NULL` if the batch is not complete, and you can retrieve
+#'   the results later by re-running `batch_chat()` when
+#'   `batch_chat_completed()` is `TRUE`.
+#' @examplesIf has_credentials("openai")
+#' chat <- chat_openai(model = "gpt-4.1-nano")
+#'
+#' # Chat ----------------------------------------------------------------------
+#'
+#' prompts <- interpolate("What do people from {{state.name}} bring to a potluck dinner?")
+#' \dontrun{
+#' chats <- batch_chat(chat, prompts, path = "potluck.json")
+#' chats
+#' }
+#'
+#' # Structured data -----------------------------------------------------------
+#' prompts <- list(
+#'   "I go by Alex. 42 years on this planet and counting.",
+#'   "Pleased to meet you! I'm Jamal, age 27.",
+#'   "They call me Li Wei. Nineteen years young.",
+#'   "Fatima here. Just celebrated my 35th birthday last week.",
+#'   "The name's Robert - 51 years old and proud of it.",
+#'   "Kwame here - just hit the big 5-0 this year."
+#' )
+#' type_person <- type_object(name = type_string(), age = type_number())
+#' \dontrun{
+#' data <- batch_chat_structured(
+#'   chat = chat,
+#'   prompts = prompts,
+#'   path = "people-data.json",
+#'   type = type_person
+#' )
+#' data
+#' }
+#'
+batch_chat <- function(chat, prompts, path, wait = TRUE) {
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    path = path,
+    wait = wait
+  )
+  job$step_until_done()
+
+  assistant_turns <- job$result_turns()
+  map2(job$user_turns, assistant_turns, function(user, assistant) {
+    if (!is.null(assistant)) {
+      chat$clone()$add_turn(user, assistant)
+    } else {
+      NULL
+    }
+  })
+}
+
+#' @export
+#' @rdname batch_chat
+#' @inheritParams parallel_chat_structured
+batch_chat_structured <- function(
+  chat,
+  prompts,
+  path,
+  type,
+  wait = TRUE,
+  convert = TRUE,
+  include_tokens = FALSE,
+  include_cost = FALSE
+) {
+  check_chat(chat)
+  provider <- chat$get_provider()
+  needs_wrapper <- S7_inherits(provider, ProviderOpenAI)
+
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    type = wrap_type_if_needed(type, needs_wrapper),
+    path = path,
+    wait = wait,
+    call = error_call
+  )
+  job$step_until_done()
+  turns <- job$result_turns()
+
+  multi_convert(
+    provider,
+    turns,
+    type,
+    convert = convert,
+    include_tokens = include_tokens,
+    include_cost = include_cost
+  )
+}
+
+#' @export
+#' @rdname batch_chat
+batch_chat_completed <- function(chat, prompts, path) {
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    path = path
+  )
+  switch(
+    job$stage,
+    "submitting" = FALSE,
+    "waiting" = !job$poll()$working,
+    "retrieving" = TRUE,
+    "done" = TRUE,
+    cli::cli_abort("Unexpected stage: {job$stage}", .internal = TRUE)
+  )
+}
+
+BatchJob <- R6::R6Class(
+  "BatchJob",
+  public = list(
+    chat = NULL,
+    user_turns = NULL,
+    path = NULL,
+    should_wait = TRUE,
+    type = NULL,
+
+    # Internal state
+    provider = NULL,
+    started_at = NULL,
+    stage = NULL,
+    batch = NULL,
+    results = NULL,
+
+    initialize = function(
+      chat,
+      prompts,
+      path,
+      type = NULL,
+      wait = TRUE,
+      call = caller_env(2)
+    ) {
+      check_chat(chat, call = call)
+      self$provider <- chat$get_provider()
+      check_has_batch_support(self$provider, call = call)
+
+      user_turns <- as_user_turns(prompts, call = call)
+      check_string(path, allow_empty = FALSE, call = call)
+      check_bool(wait, call = call)
+
+      self$chat <- chat
+      self$user_turns <- user_turns
+      self$type <- type
+      self$path <- path
+      self$should_wait <- wait
+
+      if (file.exists(path)) {
+        state <- jsonlite::read_json(path)
+        self$stage <- state$stage
+        self$batch <- state$batch
+        self$results <- state$results
+        self$started_at <- .POSIXct(state$started_at)
+
+        self$check_hash(state$hash, call = call)
+      } else {
+        self$stage <- "submitting"
+        self$batch <- NULL
+        self$started_at <- Sys.time()
+      }
+    },
+
+    save_state = function() {
+      jsonlite::write_json(
+        list(
+          version = 1,
+          stage = self$stage,
+          batch = self$batch,
+          results = self$results,
+          started_at = as.integer(self$started_at),
+          hash = self$compute_hash()
+        ),
+        self$path,
+        auto_unbox = TRUE,
+        pretty = TRUE
+      )
+    },
+
+    step = function() {
+      if (self$stage == "submitting") {
+        self$submit()
+      } else if (self$stage == "waiting") {
+        self$wait()
+      } else if (self$stage == "retrieving") {
+        self$retrieve()
+      } else {
+        cli::cli_abort("Unknown stage: {self$stage}", .internal = TRUE)
+      }
+    },
+
+    step_until_done = function() {
+      while (self$stage != "done") {
+        if (!self$step()) {
+          return(invisible())
+        }
+      }
+      invisible(self)
+    },
+
+    submit = function() {
+      existing <- self$chat$get_turns(include_system_prompt = TRUE)
+      conversations <- append_turns(list(existing), self$user_turns)
+
+      self$batch <- batch_submit(self$provider, conversations, type = self$type)
+      self$stage <- "waiting"
+      self$save_state()
+      TRUE
+    },
+
+    wait = function() {
+      # always poll once, even when wait = FALSE
+      status <- self$poll()
+
+      if (self$should_wait) {
+        cli::cli_progress_bar(
+          format = paste(
+            "{cli::pb_spin} Processing... ",
+            "{status$n_processing} -> {cli::col_green({status$n_succeeded})} / {cli::col_red({status$n_failed})} ",
+            "[{self$elapsed()}]"
path: R/batch-chat.R
user: login: gadenbuie
body: ```suggestion
            "{cli::pb_spin} Processing...",
            "[{self$elapsed()}]",
            "{status$n_processing} pending |", 
            "{cli::col_green({status$n_succeeded})} done |",
            "{cli::col_red({status$n_failed})} failed"
```

![image](https://github.com/user-attachments/assets/b22ba87d-797d-4314-adde-b7e64f08e474)

As a user, I think this format is much friendlier because I never have to guess about what the numbers mean. I do think they're _eventually_ clear, but it's nice to just not have to ask the question in the first place.
created_at: 2025-05-16T13:26:22Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,333 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` and `batch_chat_structured()` currently only work with
+#' [chat_openai()] and [chat_anthropic()]. They use the OpenAI and Anthropic
+#' batch APIs which allow you to submit multiple requests simultaneously.
+#' The results can take up to 24 hours to complete, but in return you pay 50%
+#' less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, or
+#' you're working with a different provider, you may want to use
+#' [parallel_chat()] instead.
+#'
+#' Since batched requests can take a long time to complete, `batch_chat()`
+#' requires a file path that is used to store information about the batch so
+#' you never lose any work. You can either set `wait = FALSE` or simply
+#' interrupt the waiting process, then later, either call `batch_chat()` to
+#' resume where you left off or call `batch_chat_completed()` to see if the
+#' results are ready to retrieve. Once you have the results, you can delete
+#' the file.
+#'
+#' This API is marked as experimental since I don't yet know how to handle
+#' errors in the most helpful way. Fortunately they don't seem to be common,
+#' but if you have ideas, please let me know!
+#'
+#' @inheritParams parallel_chat
+#' @param path Path to file (with `.json` extension) to store state.
+#'
+#'   The file records a hash of the provider, the prompts, and the existing
+#'   chat turns. If you attempt to reuse the same file with any of these being
+#'   different, you'll get an error.
+#' @param wait If `TRUE`, will wait for batch to complete. If `FALSE`,
+#'   it will return `NULL` if the batch is not complete, and you can retrieve
+#'   the results later by re-running `batch_chat()` when
+#'   `batch_chat_completed()` is `TRUE`.
+#' @examplesIf has_credentials("openai")
+#' chat <- chat_openai(model = "gpt-4.1-nano")
+#'
+#' # Chat ----------------------------------------------------------------------
+#'
+#' prompts <- interpolate("What do people from {{state.name}} bring to a potluck dinner?")
+#' \dontrun{
+#' chats <- batch_chat(chat, prompts, path = "potluck.json")
+#' chats
+#' }
+#'
+#' # Structured data -----------------------------------------------------------
+#' prompts <- list(
+#'   "I go by Alex. 42 years on this planet and counting.",
+#'   "Pleased to meet you! I'm Jamal, age 27.",
+#'   "They call me Li Wei. Nineteen years young.",
+#'   "Fatima here. Just celebrated my 35th birthday last week.",
+#'   "The name's Robert - 51 years old and proud of it.",
+#'   "Kwame here - just hit the big 5-0 this year."
+#' )
+#' type_person <- type_object(name = type_string(), age = type_number())
+#' \dontrun{
+#' data <- batch_chat_structured(
+#'   chat = chat,
+#'   prompts = prompts,
+#'   path = "people-data.json",
+#'   type = type_person
+#' )
+#' data
+#' }
+#'
+batch_chat <- function(chat, prompts, path, wait = TRUE) {
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    path = path,
+    wait = wait
+  )
+  job$step_until_done()
+
+  assistant_turns <- job$result_turns()
+  map2(job$user_turns, assistant_turns, function(user, assistant) {
+    if (!is.null(assistant)) {
+      chat$clone()$add_turn(user, assistant)
+    } else {
+      NULL
+    }
+  })
+}
+
+#' @export
+#' @rdname batch_chat
+#' @inheritParams parallel_chat_structured
+batch_chat_structured <- function(
+  chat,
+  prompts,
+  path,
+  type,
+  wait = TRUE,
+  convert = TRUE,
+  include_tokens = FALSE,
+  include_cost = FALSE
+) {
+  check_chat(chat)
+  provider <- chat$get_provider()
+  needs_wrapper <- S7_inherits(provider, ProviderOpenAI)
+
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    type = wrap_type_if_needed(type, needs_wrapper),
+    path = path,
+    wait = wait,
+    call = error_call
+  )
+  job$step_until_done()
+  turns <- job$result_turns()
+
+  multi_convert(
+    provider,
+    turns,
+    type,
+    convert = convert,
+    include_tokens = include_tokens,
+    include_cost = include_cost
+  )
+}
+
+#' @export
+#' @rdname batch_chat
+batch_chat_completed <- function(chat, prompts, path) {
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    path = path
+  )
+  switch(
+    job$stage,
+    "submitting" = FALSE,
+    "waiting" = !job$poll()$working,
+    "retrieving" = TRUE,
+    "done" = TRUE,
+    cli::cli_abort("Unexpected stage: {job$stage}", .internal = TRUE)
+  )
+}
+
+BatchJob <- R6::R6Class(
+  "BatchJob",
+  public = list(
+    chat = NULL,
+    user_turns = NULL,
+    path = NULL,
+    should_wait = TRUE,
+    type = NULL,
+
+    # Internal state
+    provider = NULL,
+    started_at = NULL,
+    stage = NULL,
+    batch = NULL,
+    results = NULL,
+
+    initialize = function(
+      chat,
+      prompts,
+      path,
+      type = NULL,
+      wait = TRUE,
+      call = caller_env(2)
+    ) {
+      check_chat(chat, call = call)
+      self$provider <- chat$get_provider()
+      check_has_batch_support(self$provider, call = call)
+
+      user_turns <- as_user_turns(prompts, call = call)
+      check_string(path, allow_empty = FALSE, call = call)
+      check_bool(wait, call = call)
+
+      self$chat <- chat
+      self$user_turns <- user_turns
+      self$type <- type
+      self$path <- path
+      self$should_wait <- wait
+
+      if (file.exists(path)) {
+        state <- jsonlite::read_json(path)
+        self$stage <- state$stage
+        self$batch <- state$batch
+        self$results <- state$results
+        self$started_at <- .POSIXct(state$started_at)
+
+        self$check_hash(state$hash, call = call)
+      } else {
+        self$stage <- "submitting"
+        self$batch <- NULL
+        self$started_at <- Sys.time()
+      }
+    },
+
+    save_state = function() {
+      jsonlite::write_json(
+        list(
+          version = 1,
+          stage = self$stage,
+          batch = self$batch,
+          results = self$results,
+          started_at = as.integer(self$started_at),
+          hash = self$compute_hash()
+        ),
+        self$path,
+        auto_unbox = TRUE,
+        pretty = TRUE
+      )
+    },
+
+    step = function() {
+      if (self$stage == "submitting") {
+        self$submit()
+      } else if (self$stage == "waiting") {
+        self$wait()
+      } else if (self$stage == "retrieving") {
+        self$retrieve()
+      } else {
+        cli::cli_abort("Unknown stage: {self$stage}", .internal = TRUE)
+      }
+    },
+
+    step_until_done = function() {
+      while (self$stage != "done") {
+        if (!self$step()) {
+          return(invisible())
+        }
+      }
+      invisible(self)
+    },
+
+    submit = function() {
+      existing <- self$chat$get_turns(include_system_prompt = TRUE)
+      conversations <- append_turns(list(existing), self$user_turns)
+
+      self$batch <- batch_submit(self$provider, conversations, type = self$type)
+      self$stage <- "waiting"
+      self$save_state()
+      TRUE
+    },
+
+    wait = function() {
+      # always poll once, even when wait = FALSE
+      status <- self$poll()
+
+      if (self$should_wait) {
+        cli::cli_progress_bar(
+          format = paste(
+            "{cli::pb_spin} Processing... ",
+            "{status$n_processing} -> {cli::col_green({status$n_succeeded})} / {cli::col_red({status$n_failed})} ",
+            "[{self$elapsed()}]"
+          )
+        )
+        while (status$working) {
+          Sys.sleep(0.5)
+          cli::cli_progress_update()
+          status <- self$poll()
+        }
+        cli::cli_process_done()
path: R/batch-chat.R
user: login: gadenbuie
body: ```suggestion
        cli::cli_progress_done()
```
created_at: 2025-05-16T13:26:35Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,333 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` and `batch_chat_structured()` currently only work with
+#' [chat_openai()] and [chat_anthropic()]. They use the OpenAI and Anthropic
+#' batch APIs which allow you to submit multiple requests simultaneously.
+#' The results can take up to 24 hours to complete, but in return you pay 50%
+#' less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, or
+#' you're working with a different provider, you may want to use
+#' [parallel_chat()] instead.
+#'
+#' Since batched requests can take a long time to complete, `batch_chat()`
+#' requires a file path that is used to store information about the batch so
+#' you never lose any work. You can either set `wait = FALSE` or simply
+#' interrupt the waiting process, then later, either call `batch_chat()` to
+#' resume where you left off or call `batch_chat_completed()` to see if the
+#' results are ready to retrieve. Once you have the results, you can delete
+#' the file.
+#'
+#' This API is marked as experimental since I don't yet know how to handle
+#' errors in the most helpful way. Fortunately they don't seem to be common,
+#' but if you have ideas, please let me know!
+#'
+#' @inheritParams parallel_chat
+#' @param path Path to file (with `.json` extension) to store state.
+#'
+#'   The file records a hash of the provider, the prompts, and the existing
+#'   chat turns. If you attempt to reuse the same file with any of these being
+#'   different, you'll get an error.
+#' @param wait If `TRUE`, will wait for batch to complete. If `FALSE`,
+#'   it will return `NULL` if the batch is not complete, and you can retrieve
+#'   the results later by re-running `batch_chat()` when
+#'   `batch_chat_completed()` is `TRUE`.
+#' @examplesIf has_credentials("openai")
+#' chat <- chat_openai(model = "gpt-4.1-nano")
+#'
+#' # Chat ----------------------------------------------------------------------
+#'
+#' prompts <- interpolate("What do people from {{state.name}} bring to a potluck dinner?")
+#' \dontrun{
+#' chats <- batch_chat(chat, prompts, path = "potluck.json")
+#' chats
+#' }
+#'
+#' # Structured data -----------------------------------------------------------
+#' prompts <- list(
+#'   "I go by Alex. 42 years on this planet and counting.",
+#'   "Pleased to meet you! I'm Jamal, age 27.",
+#'   "They call me Li Wei. Nineteen years young.",
+#'   "Fatima here. Just celebrated my 35th birthday last week.",
+#'   "The name's Robert - 51 years old and proud of it.",
+#'   "Kwame here - just hit the big 5-0 this year."
+#' )
+#' type_person <- type_object(name = type_string(), age = type_number())
+#' \dontrun{
+#' data <- batch_chat_structured(
+#'   chat = chat,
+#'   prompts = prompts,
+#'   path = "people-data.json",
+#'   type = type_person
+#' )
+#' data
+#' }
+#'
+batch_chat <- function(chat, prompts, path, wait = TRUE) {
path: R/batch-chat.R
user: login: hadley
body: Doh
created_at: 2025-05-16T13:30:34Z
author_association: MEMBER
diff_hunk: @@ -388,6 +388,122 @@ method(as_json, list(ProviderOpenAI, TypeObject)) <- function(provider, x) {
   )
 }
 
+
+# Batched requests -------------------------------------------------------------
+
+method(has_batch_support, ProviderOpenAI) <- function(provider) {
+  # Only enable for OpenAI, not subclasses
+  provider@name == "OpenAI"
+}
path: R/provider-openai.R
user: login: gadenbuie
body: Ah, I had wondered why `has_batch_support()` was a generic rather than by storing this as a property of `Provider` and I think this explains why.
created_at: 2025-05-16T13:36:24Z
author_association: COLLABORATOR
path: capitals.json
user: login: gadenbuie
body: I don't think you intended to commit this
created_at: 2025-05-16T13:37:47Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,37 @@
+# errors if chat/provider/prompts don't match previous run
+
+    Code
+      batch_chat(chat, prompts, path)
+    Condition
+      Error in `batch_chat()`:
+      ! provider, prompts, and user_turns don't match stored values.
path: tests/testthat/_snaps/batch-chat.md
user: login: gadenbuie
body: Should the error message include the path value to make it easier to track down the problematic file?
created_at: 2025-05-16T13:38:39Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,37 @@
+# errors if chat/provider/prompts don't match previous run
+
+    Code
+      batch_chat(chat, prompts, path)
+    Condition
+      Error in `batch_chat()`:
+      ! provider, prompts, and user_turns don't match stored values.
+      i Do you need to pick a different `path`?
+
+# informative error for bad inputs
+
+    Code
+      batch_chat("x")
+    Condition
+      Error in `batch_chat()`:
+      ! `chat` must be a <Chat> object.
+    Code
+      batch_chat(chat_ollama)
+    Condition
+      Error in `batch_chat()`:
+      ! Batch requests are not currently supported by this provider.
+    Code
+      batch_chat(chat_openai, "a")
+    Condition
+      Error in `batch_chat()`:
+      ! `prompts` must be a list or prompt, not the string "a".
path: tests/testthat/_snaps/batch-chat.md
user: login: gadenbuie
body: Is "list or prompt" expected in the error message? Or is it supposed to be "list of prompts"?
created_at: 2025-05-16T13:40:58Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,125 @@
+test_that("can get chats/data from completed request", {
+  chat <- chat_openai_test()
+
+  prompts <- list(
+    "What's the capital of Iowa?",
+    "What's the capital of New York?",
+    "What's the capital of California?",
+    "What's the capital of Texas?"
+  )
+  chats <- batch_chat(
+    chat,
+    prompts,
+    path = test_path("batch/state-capitals.json")
path: tests/testthat/test-batch-chat.R
user: login: gadenbuie
body: Should this file be committed? Or cleared out before this test?
created_at: 2025-05-16T13:43:34Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,125 @@
+test_that("can get chats/data from completed request", {
+  chat <- chat_openai_test()
+
+  prompts <- list(
+    "What's the capital of Iowa?",
+    "What's the capital of New York?",
+    "What's the capital of California?",
+    "What's the capital of Texas?"
+  )
+  chats <- batch_chat(
+    chat,
+    prompts,
+    path = test_path("batch/state-capitals.json")
+  )
+  expect_length(chats, 4)
+
+  type_state <- type_object(name = type_string("State name"))
+  data <- batch_chat_structured(
+    chat,
+    prompts,
+    path = test_path("batch/state-name.json"),
+    type = type_state
+  )
+  expect_equal(nrow(data), 4)
+})
+
+test_that("errors if chat/provider/prompts don't match previous run", {
+  chat <- chat_anthropic_test(system_prompt = "Be cool")
+  prompts <- list("What's the capital of Iowa?")
+  path <- test_path("batch/state-capitals.json")
path: tests/testthat/test-batch-chat.R
user: login: gadenbuie
body: Instead of using the file from the previous test, you could store a small JSON with enough state information to trigger the hash checks.
created_at: 2025-05-16T13:44:49Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,333 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` and `batch_chat_structured()` currently only work with
+#' [chat_openai()] and [chat_anthropic()]. They use the OpenAI and Anthropic
+#' batch APIs which allow you to submit multiple requests simultaneously.
+#' The results can take up to 24 hours to complete, but in return you pay 50%
+#' less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, or
path: R/batch-chat.R
user: login: hadley
body: Good idea!
created_at: 2025-05-16T14:11:30Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,333 @@
+#' Submit multiple chats in one batch
+#'
+#' @description
+#' `r lifecycle::badge("experimental")`
+#'
+#' `batch_chat()` and `batch_chat_structured()` currently only work with
+#' [chat_openai()] and [chat_anthropic()]. They use the OpenAI and Anthropic
+#' batch APIs which allow you to submit multiple requests simultaneously.
+#' The results can take up to 24 hours to complete, but in return you pay 50%
+#' less than usual (but note that ellmer doesn't include this discount in
+#' its pricing metadata). If you want to get results back more quickly, or
+#' you're working with a different provider, you may want to use
+#' [parallel_chat()] instead.
+#'
+#' Since batched requests can take a long time to complete, `batch_chat()`
+#' requires a file path that is used to store information about the batch so
+#' you never lose any work. You can either set `wait = FALSE` or simply
+#' interrupt the waiting process, then later, either call `batch_chat()` to
+#' resume where you left off or call `batch_chat_completed()` to see if the
+#' results are ready to retrieve. Once you have the results, you can delete
+#' the file.
+#'
+#' This API is marked as experimental since I don't yet know how to handle
+#' errors in the most helpful way. Fortunately they don't seem to be common,
+#' but if you have ideas, please let me know!
+#'
+#' @inheritParams parallel_chat
+#' @param path Path to file (with `.json` extension) to store state.
+#'
+#'   The file records a hash of the provider, the prompts, and the existing
+#'   chat turns. If you attempt to reuse the same file with any of these being
+#'   different, you'll get an error.
+#' @param wait If `TRUE`, will wait for batch to complete. If `FALSE`,
+#'   it will return `NULL` if the batch is not complete, and you can retrieve
+#'   the results later by re-running `batch_chat()` when
+#'   `batch_chat_completed()` is `TRUE`.
+#' @examplesIf has_credentials("openai")
+#' chat <- chat_openai(model = "gpt-4.1-nano")
+#'
+#' # Chat ----------------------------------------------------------------------
+#'
+#' prompts <- interpolate("What do people from {{state.name}} bring to a potluck dinner?")
+#' \dontrun{
+#' chats <- batch_chat(chat, prompts, path = "potluck.json")
+#' chats
+#' }
+#'
+#' # Structured data -----------------------------------------------------------
+#' prompts <- list(
+#'   "I go by Alex. 42 years on this planet and counting.",
+#'   "Pleased to meet you! I'm Jamal, age 27.",
+#'   "They call me Li Wei. Nineteen years young.",
+#'   "Fatima here. Just celebrated my 35th birthday last week.",
+#'   "The name's Robert - 51 years old and proud of it.",
+#'   "Kwame here - just hit the big 5-0 this year."
+#' )
+#' type_person <- type_object(name = type_string(), age = type_number())
+#' \dontrun{
+#' data <- batch_chat_structured(
+#'   chat = chat,
+#'   prompts = prompts,
+#'   path = "people-data.json",
+#'   type = type_person
+#' )
+#' data
+#' }
+#'
+batch_chat <- function(chat, prompts, path, wait = TRUE) {
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    path = path,
+    wait = wait
+  )
+  job$step_until_done()
+
+  assistant_turns <- job$result_turns()
+  map2(job$user_turns, assistant_turns, function(user, assistant) {
+    if (!is.null(assistant)) {
+      chat$clone()$add_turn(user, assistant)
+    } else {
+      NULL
+    }
+  })
+}
+
+#' @export
+#' @rdname batch_chat
+#' @inheritParams parallel_chat_structured
+batch_chat_structured <- function(
+  chat,
+  prompts,
+  path,
+  type,
+  wait = TRUE,
+  convert = TRUE,
+  include_tokens = FALSE,
+  include_cost = FALSE
+) {
+  check_chat(chat)
+  provider <- chat$get_provider()
+  needs_wrapper <- S7_inherits(provider, ProviderOpenAI)
+
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    type = wrap_type_if_needed(type, needs_wrapper),
+    path = path,
+    wait = wait,
+    call = error_call
+  )
+  job$step_until_done()
+  turns <- job$result_turns()
+
+  multi_convert(
+    provider,
+    turns,
+    type,
+    convert = convert,
+    include_tokens = include_tokens,
+    include_cost = include_cost
+  )
+}
+
+#' @export
+#' @rdname batch_chat
+batch_chat_completed <- function(chat, prompts, path) {
+  job <- BatchJob$new(
+    chat = chat,
+    prompts = prompts,
+    path = path
+  )
+  switch(
+    job$stage,
+    "submitting" = FALSE,
+    "waiting" = !job$poll()$working,
+    "retrieving" = TRUE,
+    "done" = TRUE,
+    cli::cli_abort("Unexpected stage: {job$stage}", .internal = TRUE)
+  )
+}
+
+BatchJob <- R6::R6Class(
+  "BatchJob",
+  public = list(
+    chat = NULL,
+    user_turns = NULL,
+    path = NULL,
+    should_wait = TRUE,
+    type = NULL,
+
+    # Internal state
+    provider = NULL,
+    started_at = NULL,
+    stage = NULL,
+    batch = NULL,
+    results = NULL,
+
+    initialize = function(
+      chat,
+      prompts,
+      path,
+      type = NULL,
+      wait = TRUE,
+      call = caller_env(2)
+    ) {
+      check_chat(chat, call = call)
+      self$provider <- chat$get_provider()
+      check_has_batch_support(self$provider, call = call)
+
+      user_turns <- as_user_turns(prompts, call = call)
+      check_string(path, allow_empty = FALSE, call = call)
+      check_bool(wait, call = call)
+
+      self$chat <- chat
+      self$user_turns <- user_turns
+      self$type <- type
+      self$path <- path
+      self$should_wait <- wait
+
+      if (file.exists(path)) {
+        state <- jsonlite::read_json(path)
+        self$stage <- state$stage
+        self$batch <- state$batch
+        self$results <- state$results
+        self$started_at <- .POSIXct(state$started_at)
+
+        self$check_hash(state$hash, call = call)
+      } else {
+        self$stage <- "submitting"
+        self$batch <- NULL
+        self$started_at <- Sys.time()
+      }
+    },
+
+    save_state = function() {
+      jsonlite::write_json(
+        list(
+          version = 1,
+          stage = self$stage,
+          batch = self$batch,
+          results = self$results,
+          started_at = as.integer(self$started_at),
+          hash = self$compute_hash()
+        ),
+        self$path,
+        auto_unbox = TRUE,
+        pretty = TRUE
+      )
+    },
+
+    step = function() {
+      if (self$stage == "submitting") {
+        self$submit()
+      } else if (self$stage == "waiting") {
+        self$wait()
+      } else if (self$stage == "retrieving") {
+        self$retrieve()
+      } else {
+        cli::cli_abort("Unknown stage: {self$stage}", .internal = TRUE)
+      }
+    },
+
+    step_until_done = function() {
+      while (self$stage != "done") {
+        if (!self$step()) {
+          return(invisible())
+        }
+      }
+      invisible(self)
+    },
+
+    submit = function() {
+      existing <- self$chat$get_turns(include_system_prompt = TRUE)
+      conversations <- append_turns(list(existing), self$user_turns)
+
+      self$batch <- batch_submit(self$provider, conversations, type = self$type)
+      self$stage <- "waiting"
+      self$save_state()
+      TRUE
+    },
+
+    wait = function() {
+      # always poll once, even when wait = FALSE
+      status <- self$poll()
+
+      if (self$should_wait) {
+        cli::cli_progress_bar(
+          format = paste(
+            "{cli::pb_spin} Processing... ",
+            "{status$n_processing} -> {cli::col_green({status$n_succeeded})} / {cli::col_red({status$n_failed})} ",
+            "[{self$elapsed()}]"
path: R/batch-chat.R
user: login: hadley
body: It also wraps more nicely :) 
created_at: 2025-05-16T14:13:40Z
author_association: MEMBER
path: capitals.json
user: login: hadley
body: Oops
created_at: 2025-05-16T14:16:30Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,37 @@
+# errors if chat/provider/prompts don't match previous run
+
+    Code
+      batch_chat(chat, prompts, path)
+    Condition
+      Error in `batch_chat()`:
+      ! provider, prompts, and user_turns don't match stored values.
+      i Do you need to pick a different `path`?
+
+# informative error for bad inputs
+
+    Code
+      batch_chat("x")
+    Condition
+      Error in `batch_chat()`:
+      ! `chat` must be a <Chat> object.
+    Code
+      batch_chat(chat_ollama)
+    Condition
+      Error in `batch_chat()`:
+      ! Batch requests are not currently supported by this provider.
+    Code
+      batch_chat(chat_openai, "a")
+    Condition
+      Error in `batch_chat()`:
+      ! `prompts` must be a list or prompt, not the string "a".
path: tests/testthat/_snaps/batch-chat.md
user: login: hadley
body: Hmmm, the condition is `!is.list(prompts) && !is_prompt(prompts)` so the message is technically correct albeit slightly confusing.
created_at: 2025-05-16T14:18:07Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,125 @@
+test_that("can get chats/data from completed request", {
+  chat <- chat_openai_test()
+
+  prompts <- list(
+    "What's the capital of Iowa?",
+    "What's the capital of New York?",
+    "What's the capital of California?",
+    "What's the capital of Texas?"
+  )
+  chats <- batch_chat(
+    chat,
+    prompts,
+    path = test_path("batch/state-capitals.json")
path: tests/testthat/test-batch-chat.R
user: login: hadley
body: It's deliberately committed — batch chats take too long to run to put in regular tests.
created_at: 2025-05-16T14:18:47Z
author_association: MEMBER

PR comments:
user: login: hadley
created_at: 2025-05-15T13:19:57Z
body: The problem is that you can't in store the provider on disk because it's includes credentials. And if we only stored part of the provider we'd need some relatively complicated code to stitch it back together on reload; that feels fragile to me. 

We could _maybe_ serialise the remained of the chat object to json, but that's going to be quite a bit of work, and isn't super compelling if you still have to pass in the `provider`.
author_association: MEMBER
user: login: hadley
created_at: 2025-05-15T13:29:48Z
body: BTW that is an awesome example! 
author_association: MEMBER
user: login: gadenbuie
created_at: 2025-05-15T13:33:57Z
body: > The problem is that you can't in store the provider on disk because it's includes credentials. And if we only stored part of the provider we'd need some relatively complicated code to stitch it back together on reload; that feels fragile to me.

What if it's more of a best-effort approach? Say we just store the provider name, e.g. `"provider": "openai"` or `"anthropic"` and then we call the default chat function based on that value.

If that doesn't work, `batch_chat_status()` and `batch_chat_results()` would take an optional `chat` argument where users could pass in the appropriate chat object.
author_association: COLLABORATOR
user: login: hadley
created_at: 2025-05-15T14:03:39Z
body: But that would mean that you lose the model information. Obviously, you could of course track that too, but then you're on a slippery slope. Do you think having the user re-create the chat object is really that painful?

What I like with the current design is that if your session dies for some reason, you can just re-run exactly what you ran last time:

```R
chat <- chat_openai(model = "gpt-4.1-nano")
prompts <- interpolate("What do people from {{state.name}} bring to a potluck dinner?")
chats <- batch_chat(chat, prompts, path = "potluck.json")
chats
```

But I guess you're thinking about the case where you want to launch a bunch and then check in on them later. Maybe we c could take a hybrid approach where I'd add `batch_chat_status()`, but you'd still use `batch_chat()` to actually get the results.

```R
batch_chat_status(chat, path)
```
author_association: MEMBER
user: login: t-kalinowski
created_at: 2025-05-15T14:58:03Z
body: The combination of users needing to manage the state file(s) and the lack of inspection or confirmation that reprovided prompts match those initially submitted for the state file seems like a potential footgun. I think this will lead many users to: a) unintentionally resubmit the same prompt(s) multiple times, and b) create a Chat object where the last turn responses do not correspond to the turns that were submitted.

We've discussed this extensively in the context of mall. There, we decided to create and use a cache directory where each individual completion request (i.e., a turns list) is hashed and cached. Resubmitting a completion request with identical input fetches the result from the cache. The idea is to enable users to iterate by rerunning the same script multiple times without generating a new completion request each time, retrieving cached completion responses for the same batch of inputs.

I wonder if it would be helpful to do something similar here—so that retrieval is cached locally by default and not necessarily tied to an individual response (or batch response, which the user must manually manage and be aware of).

Maybe something like this:

```r
# fetch from cache, or submit a new batch request
chat$cached_chat(..., batch = TRUE, wait = TRUE)
```

Also, I wonder how this should be integrated into the parallel chat API, as it seems to serve a very similar use case.


author_association: CONTRIBUTOR
user: login: gadenbuie
created_at: 2025-05-15T15:00:23Z
body: > But that would mean that you lose the model information.

I think we _could_ track the model information (and maybe `params`, too, if those don't carry credentials). Or we might not need to track the model – that's included in the model results (at least for OpenAI and Anthropic, but I'd imagine most would do that).

> Do you think having the user re-create the chat object is really that painful? ... you can just re-run exactly what you ran last time

I'm not thinking it's _that hard_ to recreate the `chat` object, but checking the status and getting the results only requires the provider to speak with the API. I do think we should consider it hard to recreate the prompts since batching is designed to scale up to 100k prompts. Which speaks to also including `chat$get_turns()` and `prompts` in the serialized state.

> but then you're on a slippery slope.

I can certainly see the slope and that it's slippery, but I do think it's worth serializing and storing what we can.

I guess what I'm saying is that I think `batch_chat()` should prioritize two use cases:

1. Interactive batch completions, primarily for cost reduction.
2. Entirely async batch completions that might scale to 10k completions.

In the second case, I'm thinking about the fact that your batch could take up to 24 hours to complete (likely less, but possible at scale) and that there could be days between submitting the request and retrieving the result (Anthropic gives you ~30 days to retrieve results).

Also, in the second case at scale, it seems much less likely that you're going to continue the conversation on any of the chat results, so it seems okay to me that the returned list of chats serve as a container for the completion. And if it's important to carry on the conversation you can always recreate the base chat object.
author_association: COLLABORATOR
user: login: hadley
created_at: 2025-05-15T17:08:23Z
body: @t-kalinowski @gadenbuie thanks for the feedback! Given the constraint that I am submitting ellmer to CRAN and I don't believe that this is going to be a highly used feature, I'm going to stick to the current shape of the API. I don't have time to figure out how to safely serialize and deserialize a chat/provider, and this has to be done carefully to avoid leaking credentials (and not create a maintenance headache going forward). I'm also not convinced that generating even 100k prompts is typically going to be that expensive, particularly since you will presumably want to align the LLM generated outputs with your inputs. So I think we just need to get this into the hands of users and see where their pain points are.

I will save a hash of the chat and prompts so that we can avoid the most obvious footguns. 

I do think it will be useful to have some kind of per-chat caching, but that's out of scope for ellmer, since you're going to have to much a bunch of design decisions specific for your use case.

author_association: MEMBER
user: login: gadenbuie
created_at: 2025-05-15T17:10:50Z
body: I started writing this probably at the same time as your last response. I totally agree with your conclusion, but thought I'd add this for future work/consideration

I just talked with @schloerke, and I've realized that the slope is indeed quite slippery but some of that Shiny will have to navigate to enable bookmarking existing chats, which mostly comes down to serializing `chat$get_turns()`. (And in the Shiny contexts, the user will still need to pass in a bare `chat_*()` object.) So in the long run, I'd advocate for additionally storing

1. provider name
2. model name
3. `chat$get_system_prompt()`
4. `prompts` (just text for now, maybe richer with Barret's work)
5. `chat$get_turns()` (not now, but when Barret's work is ready)

Automatically creating the `chat` object feels like a nice affordance, but after reflecting I think that requiring `chat` is a solid choice. The API could even be simplified to:

```r
# Start a batch request
batch_chat(chat, prompts, wait = FALSE)
#> Batch started. State in "ellmer-batch-abc123.json"

# Check on that batch request if you still have `prompts`
batch_chat(chat, prompts)

# Or check on it later without prompts
batch_chat(chat, path = "ellmer-batch-abc123.json")
⠙ Processing... [0:05] 46 pending | 4 done | 0 failed 

# Or get the results without prompts
chats <- batch_chat(chat, path = "ellmer-batch-abc123.json")
```
author_association: COLLABORATOR
user: login: hadley
created_at: 2025-05-15T17:18:46Z
body: @gadenbuie that reminded me to file https://github.com/tidyverse/ellmer/issues/499.

And thanks for that API sketch — I think that confirms that the current API won't be unnecessarily limiting if we save more state to disk in the future.
author_association: MEMBER
user: login: schloerke
created_at: 2025-05-15T19:34:13Z
body: Related "to JSON" record / replay PR: https://github.com/tidyverse/ellmer/pull/503/files#diff-cdd4c06a829b36da7fef0711b5523715d58b3a33abcb29c861cc95d5a8a9c1ff
author_association: COLLABORATOR
user: login: hadley
created_at: 2025-05-16T13:02:24Z
body: @gadenbuie would you mind taking one more pass at this?
author_association: MEMBER
user: login: hadley
created_at: 2025-05-16T14:19:50Z
body: @gadenbuie thanks for the thoughtful feedback, it's appreciated!
author_association: MEMBER
user: login: bshor
created_at: 2026-01-08T22:14:34Z
body: batch_chat works great for OpenAI models (haven't tried Claude). But I noticed that Google has enabled batch, not sure when. Here's [an example](https://ai.google.dev/gemini-api/docs/pricing) from it's documentation.

When you have resources, it would be great if batch_chat could be extended to Gemini models, given how well the latter are performing. No pressure, of course!
author_association: NONE
user: login: hadley
created_at: 2026-01-08T23:31:21Z
body: @bshor can you please file an issue?
author_association: MEMBER