Repository: google/gemma.cpp
Pull request #539: Add --prompt flag for non-interactive mode
Author: login: prajwalc22
Labels: name: copybara-import
description: Trigger Copybara for merging pull requests
Created: 2025-04-12T07:54:46Z | Merged: 2025-04-22T10:09:24Z

This change adds a --prompt command-line option that allows users to provide prompts directly without entering interactive mode, which is useful for scripting and automation.


Diff:
diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 00000000..ae664142
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+2b-pt-sfp.sbs filter=lfs diff=lfs merge=lfs -text
+tokenizer.spm filter=lfs diff=lfs merge=lfs -text
diff --git a/.gitignore b/.gitignore
index d4264cbe..1c13032e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,4 +1,25 @@
+# Build directories
 .cache/
 bazel-*/
 build-*/
+build/
+
+# Python cache
 python/*/__pycache__
+
+# Model files
+*.sbs
+*.spm
+*.data
+*.bin
+*.weights
+
+# IDE and editor files
+.vscode/
+.idea/
+*.swp
+*~
+
+# Local development
+.env
+.env.local
\ No newline at end of file
diff --git a/.vscode/c_cpp_properties.json b/.vscode/c_cpp_properties.json
new file mode 100644
index 00000000..64d3f903
--- /dev/null
+++ b/.vscode/c_cpp_properties.json
@@ -0,0 +1,15 @@
+{
+    "configurations": [
+        {
+            "name": "Linux",
+            "includePath": [
+                "${workspaceFolder}/**"
+            ],
+            "defines": [],
+            "cStandard": "c17",
+            "cppStandard": "c++17",
+            "intelliSenseMode": "linux-clang-x64"
+        }
+    ],
+    "version": 4
+}
\ No newline at end of file
diff --git a/CMakeLists.txt b/CMakeLists.txt
index b5728350..b9558ad1 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-cmake_minimum_required(VERSION 3.11)
+cmake_minimum_required(VERSION 3.11...4.0)
 
 include(FetchContent)
 
diff --git a/build/.gitignore b/build/.gitignore
deleted file mode 100644
index 3822a0b1..00000000
--- a/build/.gitignore
+++ /dev/null
@@ -1,3 +0,0 @@
-*
-!.gitignore
-!.hgignore
\ No newline at end of file
diff --git a/gemma/gemma_args.h b/gemma/gemma_args.h
index 4fe2d332..d02decec 100644
--- a/gemma/gemma_args.h
+++ b/gemma/gemma_args.h
@@ -28,10 +28,10 @@
 #include "compression/shared.h"
 #include "gemma/common.h"
 #include "gemma/gemma.h"  // For CreateGemma
+#include "hwy/base.h"     // HWY_ABORT
 #include "ops/matmul.h"
 #include "util/args.h"
 #include "util/basics.h"  // Tristate
-#include "hwy/base.h"       // HWY_ABORT
 
 namespace gcpp {
 
@@ -106,9 +106,8 @@ struct LoaderArgs : public ArgsBase<LoaderArgs> {
             "Path name of model weights (.sbs) file.\n  Required argument.\n");
     visitor(compressed_weights, "compressed_weights", Path(),
             "Deprecated alias for --weights.");
-    visitor(
-        model_type_str, "model", std::string(),
-        "Model type, see common.cc for valid values.\n");
+    visitor(model_type_str, "model", std::string(),
+            "Model type, see common.cc for valid values.\n");
     visitor(weight_type_str, "weight_type", std::string("sfp"),
             "Weight type\n    f32 = float, bf16 = bfloat16, sfp = 8-bit SFP.");
   }
@@ -117,8 +116,6 @@ struct LoaderArgs : public ArgsBase<LoaderArgs> {
   const ModelInfo& Info() const { return info_; }
 
  private:
-  // TODO(rays): remove this. Eventually ModelConfig will be loaded from the
-  // weights file, so we can remove the need for this struct entirely.
   ModelInfo info_;
 };
 
@@ -161,6 +158,7 @@ struct InferenceArgs : public ArgsBase<InferenceArgs> {
   bool multiturn;
   Path image_file;
 
+  std::string prompt;  // Added prompt flag for non-interactive mode
   std::string eot_line;
 
   // Returns error string or nullptr if OK.
@@ -178,7 +176,7 @@ struct InferenceArgs : public ArgsBase<InferenceArgs> {
             "Show verbose developer information\n    0 = only print generation "
             "output\n    1 = standard user-facing terminal ui\n    2 = show "
             "developer/debug info).\n    Default = 1.",
-            2);
+            1);  // Changed verbosity level to 1 since it's user-facing
 
     visitor(max_generated_tokens, "max_generated_tokens", size_t{2048},
             "Maximum number of tokens to generate.");
@@ -200,6 +198,12 @@ struct InferenceArgs : public ArgsBase<InferenceArgs> {
             "resets every turn)");
     visitor(image_file, "image_file", Path(), "Image file to load.");
 
+    visitor(prompt, "prompt", std::string(""),
+            "Initial prompt for non-interactive mode. When specified, "
+            "generates a response"
+            " and exits.",
+            1);  // Added as user-facing option
+
     visitor(
         eot_line, "eot_line", std::string(""),
         "End of turn line. "
@@ -233,4 +237,4 @@ struct InferenceArgs : public ArgsBase<InferenceArgs> {
 
 }  // namespace gcpp
 
-#endif  // THIRD_PARTY_GEMMA_CPP_GEMMA_ARGS_H_
+#endif  // THIRD_PARTY_GEMMA_CPP_GEMMA_ARGS_H_
\ No newline at end of file
diff --git a/gemma/run.cc b/gemma/run.cc
index 5170b6e9..2de1c1d6 100644
--- a/gemma/run.cc
+++ b/gemma/run.cc
@@ -27,13 +27,13 @@
 #include "evals/benchmark_helper.h"
 #include "gemma/common.h"
 #include "gemma/gemma.h"  // Gemma
-#include "gemma/gemma_args.h"  // LoaderArgs
-#include "ops/matmul.h"        // MatMulEnv
-#include "paligemma/image.h"
-#include "util/args.h"  // HasHelp
-#include "util/threading_context.h"
+#include "gemma/gemma_args.h"
+#include "hwy/base.h"
 #include "hwy/highway.h"
 #include "hwy/profiler.h"
+#include "ops/matmul.h"  // MatMulEnv
+#include "paligemma/image.h"
+#include "util/args.h"  // HasHelp
 
 #if (!defined(HWY_VERSION_LT) || HWY_VERSION_LT(1, 2)) && !HWY_IDE
 #error "Please update to version 1.2 of github.com/google/highway."
@@ -77,6 +77,17 @@ std::string GetPrompt(std::istream& input, int verbosity,
   return prompt_string;
 }
 
+// Get prompt either from interactive input or command line
+std::string GetPrompt(const InferenceArgs& inference) {
+  // If prompt is provided via command line, use that
+  if (!inference.prompt.empty()) {
+    return inference.prompt;
+  }
+
+  // Otherwise get interactive prompt
+  return GetPrompt(std::cin, inference.verbosity, inference.eot_line);
+}
+
 // The main Read-Eval-Print Loop.
 void ReplGemma(const ThreadingArgs& threading, const InferenceArgs& inference,
                Gemma& model, KVCache& kv_cache) {
@@ -149,18 +160,21 @@ void ReplGemma(const ThreadingArgs& threading, const InferenceArgs& inference,
     tokens_generated_this_turn = 0;
 
     // Read prompt and handle special commands.
-    std::string prompt_string =
-        GetPrompt(std::cin, inference.verbosity, inference.eot_line);
-    if (!std::cin) return;
+    std::string prompt_string = GetPrompt(inference);
+
+    if (!std::cin && inference.prompt.empty()) return;
+
     // If !eot_line.empty(), we append \n, so only look at the first 2 chars.
-    if (prompt_string.size() >= 2 && prompt_string[0] == '%') {
+    if (inference.prompt.empty() && prompt_string.size() >= 2 &&
+        prompt_string[0] == '%') {
       if (prompt_string[1] == 'q' || prompt_string[1] == 'Q') return;
       if (prompt_string[1] == 'c' || prompt_string[1] == 'C') {
         abs_pos = 0;
         continue;
       }
     }
-    if (prompt_string.empty()) {
+
+    if (inference.prompt.empty() && prompt_string.empty()) {
       std::cout << "Use '%q' to quit.\n";
       continue;
     }
@@ -172,9 +186,9 @@ void ReplGemma(const ThreadingArgs& threading, const InferenceArgs& inference,
                                     .stream_token = stream_token,
                                     .use_spinning = threading.spin};
     inference.CopyTo(runtime_config);
-    size_t prefix_end = 0;
-
     std::vector<int> prompt;
+    size_t prompt_size = 0;
+    size_t prefix_end = 0;
     if (have_image) {
       prompt =
           WrapAndTokenize(model.Tokenizer(), model.ChatTemplate(), model.Info(),
@@ -184,8 +198,9 @@ void ReplGemma(const ThreadingArgs& threading, const InferenceArgs& inference,
       // The end of the prefix for prefix-LM style attention in Paligemma.
       // See Figure 2 of https://arxiv.org/abs/2407.07726.
       prefix_end = prompt_size;
-      // We need to look at all the tokens for the prefix.
-      runtime_config.prefill_tbatch_size = prompt_size;
+
+      // REMOVED: Don't change prefill_tbatch_size for image handling
+      // runtime_config.prefill_tbatch_size = prompt_size;
     } else {
       prompt = WrapAndTokenize(model.Tokenizer(), model.ChatTemplate(),
                                model.Info(), abs_pos, prompt_string);
@@ -206,6 +221,11 @@ void ReplGemma(const ThreadingArgs& threading, const InferenceArgs& inference,
                    timing_info);
     std::cout << "\n\n";
 
+    // Break the loop if in non-interactive mode
+    if (!inference.prompt.empty()) {
+      break;
+    }
+
     // Prepare for the next turn. Works only for PaliGemma.
     if (!inference.multiturn ||
         model.Info().wrapping == PromptWrapping::PALIGEMMA) {
@@ -259,10 +279,13 @@ void Run(ThreadingArgs& threading, LoaderArgs& loader,
     instructions += multiturn;
     instructions += examples;
 
-    std::cout << "\033[2J\033[1;1H"  // clear screen
-              << kAsciiArtBanner << "\n\n";
-    ShowConfig(threading, loader, inference);
-    std::cout << "\n" << instructions << "\n";
+    // Skip the banner and instructions in non-interactive mode
+    if (inference.prompt.empty()) {
+      std::cout << "\033[2J\033[1;1H"  // clear screen
+                << kAsciiArtBanner << "\n\n";
+      ShowConfig(threading, loader, inference);
+      std::cout << "\n" << instructions << "\n";
+    }
   }
 
   ReplGemma(threading, inference, model, kv_cache);
@@ -280,6 +303,7 @@ int main(int argc, char** argv) {
 
     if (gcpp::HasHelp(argc, argv)) {
       std::cerr << gcpp::kAsciiArtBanner;
+
       gcpp::ShowHelp(threading, loader, inference);
       return 0;
     }
@@ -300,4 +324,4 @@ int main(int argc, char** argv) {
   }
   PROFILER_PRINT_RESULTS();  // Must call outside the zone above.
   return 0;
-}
+}
\ No newline at end of file


Reviews:
user: login: jan-wassenberg
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-16T05:50:17Z
user: login: prajwalc22
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T07:10:52Z
user: login: prajwalc22
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T07:12:26Z
user: login: jan-wassenberg
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T10:56:37Z
user: login: jan-wassenberg
body: Thanks for making the changes. Almost there :)
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-16T12:02:56Z
user: login: jan-wassenberg
body: Thanks for making the changes. Nice!
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-04-17T12:53:19Z
user: login: jan-wassenberg
body: Thanks for updating :)
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-22T07:58:08Z

Review comments:
diff_hunk: @@ -1,583 +1,27 @@
-# gemma.cpp
+---
+library_name: gemma.cpp
path: README.md
user: login: jan-wassenberg
body: A model card can be useful, but let's not overwrite the previous README.md with it :) Please revert this file, we can add another file if you like.
created_at: 2025-04-16T05:38:24Z
author_association: MEMBER
diff_hunk: @@ -28,13 +28,205 @@
 #include "compression/shared.h"
 #include "gemma/common.h"
 #include "gemma/gemma.h"  // For CreateGemma
+#include "hwy/base.h"     // HWY_IS_ASAN, HWY_ABORT
 #include "ops/matmul.h"
+#include "util/allocator.h"
 #include "util/args.h"
 #include "util/basics.h"  // Tristate
-#include "hwy/base.h"       // HWY_ABORT
+#include "util/threading.h"
+#include "util/threading_context.h"
 
 namespace gcpp {
 
+static inline const char* CompiledConfig() {
+  if (HWY_IS_ASAN) {
+    return "asan";
+  } else if (HWY_IS_MSAN) {
+    return "msan";
+  } else if (HWY_IS_TSAN) {
+    return "tsan";
+  } else if (HWY_IS_HWASAN) {
+    return "hwasan";
+  } else if (HWY_IS_UBSAN) {
+    return "ubsan";
+  } else if (HWY_IS_DEBUG_BUILD) {
+    return "dbg";
+  } else {
+    return "opt";
+  }
+}
+template <typename Derived>
+struct ArgsBase {
path: gemma/gemma_args.h
user: login: jan-wassenberg
body: hm, this was/still is defined in util/args.h, is it necessary to change the base class for adding the flag?
If you want to improve the ArgsBase, we can overwrite the one in util/args.h instead?
created_at: 2025-04-16T05:42:01Z
author_association: MEMBER
diff_hunk: @@ -28,13 +28,205 @@
 #include "compression/shared.h"
 #include "gemma/common.h"
 #include "gemma/gemma.h"  // For CreateGemma
+#include "hwy/base.h"     // HWY_IS_ASAN, HWY_ABORT
 #include "ops/matmul.h"
+#include "util/allocator.h"
 #include "util/args.h"
 #include "util/basics.h"  // Tristate
-#include "hwy/base.h"       // HWY_ABORT
+#include "util/threading.h"
+#include "util/threading_context.h"
 
 namespace gcpp {
 
+static inline const char* CompiledConfig() {
+  if (HWY_IS_ASAN) {
+    return "asan";
+  } else if (HWY_IS_MSAN) {
+    return "msan";
+  } else if (HWY_IS_TSAN) {
+    return "tsan";
+  } else if (HWY_IS_HWASAN) {
+    return "hwasan";
+  } else if (HWY_IS_UBSAN) {
+    return "ubsan";
+  } else if (HWY_IS_DEBUG_BUILD) {
+    return "dbg";
+  } else {
+    return "opt";
+  }
+}
+template <typename Derived>
+struct ArgsBase {
+  void Init() { static_cast<Derived*>(this)->ForEach(SetToDefault()); }
+
+  void InitAndParse(int argc, char* argv[]) {
+    Init();
+    static_cast<Derived*>(this)->ForEach(ParseOption(argc, argv));
+  }
+
+  void Print(int min_verbosity = 1) const {
+    static_cast<const Derived*>(this)->ForEach(PrintOption(min_verbosity));
+  }
+
+  void Help() const { static_cast<const Derived*>(this)->ForEach(PrintHelp()); }
+
+ protected:
+  // Helper struct for printing help messages
+  struct PrintHelp {
+    template <typename T>
+    void operator()(const T& value, const char* name, const T& default_value,
+                    const char* description, int verbosity = 1) const {
+      fprintf(stderr, "  --%s\n        %s\n", name, description);
+    }
+    // Special case for strings to avoid template deduction issues
+    void operator()(const std::string& value, const char* name,
+                    const std::string& default_value, const char* description,
+                    int verbosity = 1) const {
+      fprintf(stderr, "  --%s\n        %s\n", name, description);
+    }
+    // Special case for Path type
+    void operator()(const Path& value, const char* name,
+                    const Path& default_value, const char* description,
+                    int verbosity = 1) const {
+      fprintf(stderr, "  --%s\n        %s\n", name, description);
+    }
+  };
+
+  // Helper struct for setting default values
+  struct SetToDefault {
+    template <typename T>
+    void operator()(T& value, const char* name, const T& default_value,
+                    const char* description, int verbosity = 1) const {
+      value = default_value;
+    }
+  };
+
+  // Helper struct for printing values
+  struct PrintOption {
+    explicit PrintOption(int min_verbosity) : min_verbosity_(min_verbosity) {}
+
+    template <typename T>
+    void operator()(const T& value, const char* name, const T& default_value,
+                    const char* description, int verbosity = 1) const {
+      if (verbosity >= min_verbosity_) {
+        fprintf(stderr, "%s: %s\n", name, ToString(value).c_str());
+      }
+    }
+
+   private:
+    int min_verbosity_;
+
+    // Helper function to convert values to string
+    template <typename T>
+    static std::string ToString(const T& value) {
+      return std::to_string(value);
+    }
+    // Specialization for string
+    static std::string ToString(const std::string& value) { return value; }
+    // Specialization for Path
+    static std::string ToString(const Path& value) { return value.path; }
+  };
+};
+struct ThreadingArgs : public ArgsBase<ThreadingArgs> {
+ public:
+  ThreadingArgs(int argc, char* argv[]) { InitAndParse(argc, argv); }
+  ThreadingArgs() { Init(); };
+
+  int verbosity;
+
+  size_t max_threads;  // divided among the detected clusters
+  Tristate pin;        // pin threads?
+  Tristate spin;       // use spin waits?
+
+  // For BoundedSlice:
+  size_t skip_packages;
+  size_t max_packages;
+  size_t skip_clusters;
+  size_t max_clusters;
+  size_t skip_lps;
+  size_t max_lps;
+
+  std::string eot_line;
+  std::string prompt;
+  template <class Visitor>
+  void ForEach(const Visitor& visitor) {
+    visitor(verbosity, "verbosity", 1,
+            "Show verbose developer information\n    0 = only print generation "
+            "output\n    1 = standard user-facing terminal ui\n    2 = show "
+            "developer/debug info).\n    Default = 1.",
+            2);
+
+    // The exact meaning is more subtle: see the comment at NestedPools ctor.
+    visitor(max_threads, "num_threads", size_t{0},
+            "Maximum number of threads to use; default 0 = unlimited.", 2);
+    visitor(pin, "pin", Tristate::kDefault,
+            "Pin threads? -1 = auto, 0 = no, 1 = yes.", 2);
+    visitor(spin, "spin", Tristate::kDefault,
+            "Use spin waits? -1 = auto, 0 = no, 1 = yes.", 2);
+    // These can be used to partition CPU sockets/packages and their
+    // clusters/CCXs across several program instances. The default is to use
+    // all available resources.
+    visitor(skip_packages, "skip_packages", size_t{0},
+            "Index of the first socket to use; default 0 = unlimited.", 2);
+    visitor(max_packages, "max_packages", size_t{0},
+            "Maximum number of sockets to use; default 0 = unlimited.", 2);
+    visitor(skip_clusters, "skip_clusters", size_t{0},
+            "Index of the first CCX to use; default 0 = unlimited.", 2);
+    visitor(max_clusters, "max_clusters", size_t{0},
+            "Maximum number of CCXs to use; default 0 = unlimited.", 2);
+    // These are only used when CPU topology is unknown.
+    visitor(skip_lps, "skip_lps", size_t{0},
+            "Index of the first LP to use; default 0 = unlimited.", 2);
+    visitor(max_lps, "max_lps", size_t{0},
+            "Maximum number of LPs to use; default 0 = unlimited.", 2);
+
+    visitor(
+        eot_line, "eot_line", std::string(""),
+        "End of turn line. "
+        "When you specify this, the prompt will be all lines "
+        "before the line where only the given string appears.\n    Default = "
+        "When a newline is encountered, that signals the end of the turn.",
+        2);
+
+    visitor(prompt, "prompt", std::string(""),
path: gemma/gemma_args.h
user: login: jan-wassenberg
body: The new prompt flag seems a better fit for InferenceArgs? BTW I'm curious where you got ThreadingArgs? This is newly written code from a pull request. I would take care of integrating it when that PR lands, no need to copy it now/before then :)
created_at: 2025-04-16T05:45:25Z
author_association: MEMBER
diff_hunk: @@ -238,6 +249,22 @@ void Run(ThreadingArgs& threading, LoaderArgs& loader,
   KVCache kv_cache =
       KVCache::Create(model.GetModelConfig(), inference.prefill_tbatch_size);
 
+  if (!threading.prompt.empty()) {
path: gemma/run.cc
user: login: jan-wassenberg
body: I'd prefer not to duplicate this code, which increases maintenance burden. Instead we can re-use the existing ReplGemma, and pass to GetPrompt the inference args (rather than individual fields), and check inside GetPrompt if prompt is not empty, and if so return that immediately.
Then to exit, we could introduce a flag outside of the `while(true)` loop that checks whether prompt was not empty, and if so, exit the loop. This adds another condition next to the existing `prompt_string[1] == 'q'`. What do you think?
created_at: 2025-04-16T05:50:10Z
author_association: MEMBER
diff_hunk: @@ -238,6 +249,22 @@ void Run(ThreadingArgs& threading, LoaderArgs& loader,
   KVCache kv_cache =
       KVCache::Create(model.GetModelConfig(), inference.prefill_tbatch_size);
 
+  if (!threading.prompt.empty()) {
path: gemma/run.cc
user: login: prajwalc22
body: Thank you for the review! I'll make the following changes:

1. README.md changes:
   - I'll revert the README.md changes back to original
   - Create a separate MODEL_CARD.md for the model documentation

2. ArgsBase implementation:
   - I'll remove the duplicate ArgsBase implementation from gemma_args.h since it's already in util/args.h

3. Prompt flag location:
   - I'll move the prompt flag from ThreadingArgs to InferenceArgs where it fits better logically
   - Will update verbosity level to 1 since it's a user-facing option

4. run.cc simplification:
   - I'll modify the code to reuse the existing ReplGemma implementation
   - Will implement the suggested approach of checking prompt in GetPrompt and adding a loop exit condition for non-interactive mode
   - Remove the separate code path for non-interactive mode

 I'll push these changes soon.
created_at: 2025-04-16T07:10:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -28,13 +28,205 @@
 #include "compression/shared.h"
 #include "gemma/common.h"
 #include "gemma/gemma.h"  // For CreateGemma
+#include "hwy/base.h"     // HWY_IS_ASAN, HWY_ABORT
 #include "ops/matmul.h"
+#include "util/allocator.h"
 #include "util/args.h"
 #include "util/basics.h"  // Tristate
-#include "hwy/base.h"       // HWY_ABORT
+#include "util/threading.h"
+#include "util/threading_context.h"
 
 namespace gcpp {
 
+static inline const char* CompiledConfig() {
+  if (HWY_IS_ASAN) {
+    return "asan";
+  } else if (HWY_IS_MSAN) {
+    return "msan";
+  } else if (HWY_IS_TSAN) {
+    return "tsan";
+  } else if (HWY_IS_HWASAN) {
+    return "hwasan";
+  } else if (HWY_IS_UBSAN) {
+    return "ubsan";
+  } else if (HWY_IS_DEBUG_BUILD) {
+    return "dbg";
+  } else {
+    return "opt";
+  }
+}
+template <typename Derived>
+struct ArgsBase {
+  void Init() { static_cast<Derived*>(this)->ForEach(SetToDefault()); }
+
+  void InitAndParse(int argc, char* argv[]) {
+    Init();
+    static_cast<Derived*>(this)->ForEach(ParseOption(argc, argv));
+  }
+
+  void Print(int min_verbosity = 1) const {
+    static_cast<const Derived*>(this)->ForEach(PrintOption(min_verbosity));
+  }
+
+  void Help() const { static_cast<const Derived*>(this)->ForEach(PrintHelp()); }
+
+ protected:
+  // Helper struct for printing help messages
+  struct PrintHelp {
+    template <typename T>
+    void operator()(const T& value, const char* name, const T& default_value,
+                    const char* description, int verbosity = 1) const {
+      fprintf(stderr, "  --%s\n        %s\n", name, description);
+    }
+    // Special case for strings to avoid template deduction issues
+    void operator()(const std::string& value, const char* name,
+                    const std::string& default_value, const char* description,
+                    int verbosity = 1) const {
+      fprintf(stderr, "  --%s\n        %s\n", name, description);
+    }
+    // Special case for Path type
+    void operator()(const Path& value, const char* name,
+                    const Path& default_value, const char* description,
+                    int verbosity = 1) const {
+      fprintf(stderr, "  --%s\n        %s\n", name, description);
+    }
+  };
+
+  // Helper struct for setting default values
+  struct SetToDefault {
+    template <typename T>
+    void operator()(T& value, const char* name, const T& default_value,
+                    const char* description, int verbosity = 1) const {
+      value = default_value;
+    }
+  };
+
+  // Helper struct for printing values
+  struct PrintOption {
+    explicit PrintOption(int min_verbosity) : min_verbosity_(min_verbosity) {}
+
+    template <typename T>
+    void operator()(const T& value, const char* name, const T& default_value,
+                    const char* description, int verbosity = 1) const {
+      if (verbosity >= min_verbosity_) {
+        fprintf(stderr, "%s: %s\n", name, ToString(value).c_str());
+      }
+    }
+
+   private:
+    int min_verbosity_;
+
+    // Helper function to convert values to string
+    template <typename T>
+    static std::string ToString(const T& value) {
+      return std::to_string(value);
+    }
+    // Specialization for string
+    static std::string ToString(const std::string& value) { return value; }
+    // Specialization for Path
+    static std::string ToString(const Path& value) { return value.path; }
+  };
+};
+struct ThreadingArgs : public ArgsBase<ThreadingArgs> {
+ public:
+  ThreadingArgs(int argc, char* argv[]) { InitAndParse(argc, argv); }
+  ThreadingArgs() { Init(); };
+
+  int verbosity;
+
+  size_t max_threads;  // divided among the detected clusters
+  Tristate pin;        // pin threads?
+  Tristate spin;       // use spin waits?
+
+  // For BoundedSlice:
+  size_t skip_packages;
+  size_t max_packages;
+  size_t skip_clusters;
+  size_t max_clusters;
+  size_t skip_lps;
+  size_t max_lps;
+
+  std::string eot_line;
+  std::string prompt;
+  template <class Visitor>
+  void ForEach(const Visitor& visitor) {
+    visitor(verbosity, "verbosity", 1,
+            "Show verbose developer information\n    0 = only print generation "
+            "output\n    1 = standard user-facing terminal ui\n    2 = show "
+            "developer/debug info).\n    Default = 1.",
+            2);
+
+    // The exact meaning is more subtle: see the comment at NestedPools ctor.
+    visitor(max_threads, "num_threads", size_t{0},
+            "Maximum number of threads to use; default 0 = unlimited.", 2);
+    visitor(pin, "pin", Tristate::kDefault,
+            "Pin threads? -1 = auto, 0 = no, 1 = yes.", 2);
+    visitor(spin, "spin", Tristate::kDefault,
+            "Use spin waits? -1 = auto, 0 = no, 1 = yes.", 2);
+    // These can be used to partition CPU sockets/packages and their
+    // clusters/CCXs across several program instances. The default is to use
+    // all available resources.
+    visitor(skip_packages, "skip_packages", size_t{0},
+            "Index of the first socket to use; default 0 = unlimited.", 2);
+    visitor(max_packages, "max_packages", size_t{0},
+            "Maximum number of sockets to use; default 0 = unlimited.", 2);
+    visitor(skip_clusters, "skip_clusters", size_t{0},
+            "Index of the first CCX to use; default 0 = unlimited.", 2);
+    visitor(max_clusters, "max_clusters", size_t{0},
+            "Maximum number of CCXs to use; default 0 = unlimited.", 2);
+    // These are only used when CPU topology is unknown.
+    visitor(skip_lps, "skip_lps", size_t{0},
+            "Index of the first LP to use; default 0 = unlimited.", 2);
+    visitor(max_lps, "max_lps", size_t{0},
+            "Maximum number of LPs to use; default 0 = unlimited.", 2);
+
+    visitor(
+        eot_line, "eot_line", std::string(""),
+        "End of turn line. "
+        "When you specify this, the prompt will be all lines "
+        "before the line where only the given string appears.\n    Default = "
+        "When a newline is encountered, that signals the end of the turn.",
+        2);
+
+    visitor(prompt, "prompt", std::string(""),
path: gemma/gemma_args.h
user: login: prajwalc22
body: You're right about ThreadingArgs 
I apologize for the confusion. While resolving some conflicts, I noticed this code and incorrectly assumed it was part of the main codebase. I'll remove it and keep my changes focused on just the --prompt functionality.

created_at: 2025-04-16T07:12:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -238,6 +249,22 @@ void Run(ThreadingArgs& threading, LoaderArgs& loader,
   KVCache kv_cache =
       KVCache::Create(model.GetModelConfig(), inference.prefill_tbatch_size);
 
+  if (!threading.prompt.empty()) {
path: gemma/run.cc
user: login: jan-wassenberg
body: Sounds good, thanks!
created_at: 2025-04-16T10:56:37Z
author_association: MEMBER
diff_hunk: @@ -149,21 +161,32 @@ void ReplGemma(const ThreadingArgs& threading, const InferenceArgs& inference,
     tokens_generated_this_turn = 0;
 
     // Read prompt and handle special commands.
-    std::string prompt_string =
-        GetPrompt(std::cin, inference.verbosity, inference.eot_line);
-    if (!std::cin) return;
+    std::string prompt_string = GetPrompt(inference);
+
+    if (!std::cin && inference.prompt.empty()) return;
+
     // If !eot_line.empty(), we append \n, so only look at the first 2 chars.
-    if (prompt_string.size() >= 2 && prompt_string[0] == '%') {
+    if (inference.prompt.empty() && prompt_string.size() >= 2 &&
+        prompt_string[0] == '%') {
       if (prompt_string[1] == 'q' || prompt_string[1] == 'Q') return;
       if (prompt_string[1] == 'c' || prompt_string[1] == 'C') {
         abs_pos = 0;
         continue;
       }
     }
-    if (prompt_string.empty()) {
+
+    if (inference.prompt.empty() && prompt_string.empty()) {
       std::cout << "Use '%q' to quit.\n";
       continue;
     }
+    std::vector<int> prompt;
+    size_t prompt_size = 0;
+    size_t prefix_end = 0;
+    if constexpr (kVerboseLogTokens) {
path: gemma/run.cc
user: login: jan-wassenberg
body: Looks like this will always be empty. Let's move it back after prompt has been set?
created_at: 2025-04-16T12:00:08Z
author_association: MEMBER
diff_hunk: @@ -192,10 +211,13 @@ void ReplGemma(const ThreadingArgs& threading, const InferenceArgs& inference,
       prompt_size = prompt.size();
     }
 
-    if constexpr (kVerboseLogTokens) {
-      for (int i = 0; i < prompt_size; ++i) {
-        fprintf(stderr, "DDD TOKEN %3d: %6d\n", i, prompt[i]);
-      }
+    if (have_image) {
+      runtime_config.image_tokens = &image_tokens;
+      prompt_size = prompt.size();
+      // The end of the prefix for prefix-LM style attention in Paligemma.
+      prefix_end = prompt_size;
+      // We need to look at all the tokens for the prefix.
+      runtime_config.prefill_tbatch_size = prompt_size;
path: gemma/run.cc
user: login: jan-wassenberg
body: Why this change? It is OK to have smaller batches than the prompt. For prompts > 512 or so, we want to break them up into multiple batches. I think we can leave prefill_tbatch_size unchanged, no?
created_at: 2025-04-16T12:02:21Z
author_association: MEMBER

PR comments:
user: login: jan-wassenberg
created_at: 2025-04-14T14:25:20Z
body: Thanks for adding, this is nice to have. We are only able to merge into the dev branch, would you mind rebasing/replacing your PR to there?
author_association: MEMBER
user: login: prajwalc22
created_at: 2025-04-15T04:13:39Z
body: Thanks for the update! I attempted to rebase onto the dev branch, but I ran into some conflicts. I’ll work on resolving them and update the PR shortly.
author_association: CONTRIBUTOR
user: login: jan-wassenberg
created_at: 2025-04-15T11:29:44Z
body: Yes, sorry about the merge conflict. We recently cleaned up the interface a bit. More on the way related to weights handling, but it will take a few more days to land.
Sounds good!
author_association: MEMBER
user: login: prajwalc22
created_at: 2025-04-16T04:40:03Z
body:  I've successfully rebased the changes onto the dev branch and resolved the conflicts. The PR is now ready for review.  Would you be able to approve the workflow run when you have a chance? Thank you in advance!
author_association: CONTRIBUTOR
user: login: prajwalc22
created_at: 2025-04-16T11:05:10Z
body: I've addressed all the requested changes. Could you please review the changes?
 Let me know if any further adjustments are needed :)
author_association: CONTRIBUTOR
user: login: prajwalc22
created_at: 2025-04-17T04:47:09Z
body: @jan-wassenberg I've addressed all the feedback from the review:

1. Removed the change to prefill_tbatch_size in the image handling code - it's no longer being set to prompt_size
2. Fixed the variable placement issue by moving the prompt vector initialization to after the prompt_string is set
3. Simplified the non-interactive mode logic
4. Maintained the prompt flag in InferenceArgs as requested

Ready for another review, thanks for your feedback!
author_association: CONTRIBUTOR
user: login: prajwalc22
created_at: 2025-04-17T13:51:29Z
body: Thank you for the review and approval! :)

author_association: CONTRIBUTOR
user: login: jan-wassenberg
created_at: 2025-04-17T17:01:02Z
body: :) YW, thanks for implementing it!
CI has found an issue: the new include of threading.h in run.cc requires a dep be added to bazel BUILD, or preferably just removing the include, which I think is unnecessary.
FYI I will be out for Easter until Tue.
author_association: MEMBER
user: login: prajwalc22
created_at: 2025-04-17T18:01:26Z
body:  I'll remove the unnecessary threading.h include from run.cc  implementation. I'll make that change right away.

Enjoy your Easter break! :)
See you  when you're back on Tuesday.
author_association: CONTRIBUTOR