Repository: ggml-org/llama.cpp
Pull request: Callback before abort

Summary: Add a callback that will be called just before abort. This allows apps without a console to display a message to the user and save data if needed.

I have created a simple patch that adds a callback to be called before fatal errors, which currently lead to abort. This is very useful in case llama.cpp is run without a console, because llama.cpp will attempt to print a message to console, but for apps without a console, that message will obviously be lost. Instead, the callback allows the application to show a message box, save state, or do similar things before the forced abort. I hope you can accept the pull request.

There is a possible future improvement to be discussed: the callback could return a value in order to veto the forced application shutdown. In this case, llama.cpp should immediately stop processing instead of aborting the whole process. llama.cpp functions should free data, then exit immediately and return control to the calling application as soon as possible. This would allow users to free additional RAM or change some settings to hopefully allow the next promt to be processed without problems. An implementation could introduce a global variable, like ggml_abort, to stop all further processing in llama.cpp. This would require checking the variable at central points of processing. A more complicated approach would be changing all functions to return error information and checking that information from all functions called.

A similar request was made here, but it seems to take a different direction: https://github.com/ggml-org/ggml/issues/1083


Changed files:
filename: ggml/include/ggml.h
patch: @@ -314,6 +314,13 @@
 extern "C" {
 #endif
 
+    // Function type used in fatal error callbacks
+    typedef void (*ggml_abort_callback_t)(const char * error_message);
+
+    // Set the abort callback (passing null will restore original abort functionality: printing a message to stdout)
+    // Returns the old callback for chaining
+    GGML_API ggml_abort_callback_t ggml_set_abort_callback(ggml_abort_callback_t callback);
+
     GGML_NORETURN GGML_ATTRIBUTE_FORMAT(3, 4)
     GGML_API void ggml_abort(const char * file, int line, const char * fmt, ...);
 

filename: ggml/src/ggml.c
patch: @@ -202,19 +202,34 @@ void ggml_print_backtrace(void) {
 }
 #endif
 
+static ggml_abort_callback_t g_abort_callback = NULL;
+
+// Set the abort callback (passing null will restore original abort functionality: printing a message to stdout)
+GGML_API ggml_abort_callback_t ggml_set_abort_callback(ggml_abort_callback_t callback) {
+    ggml_abort_callback_t ret_val = g_abort_callback;
+    g_abort_callback = callback;
+    return ret_val;
+}
+
 void ggml_abort(const char * file, int line, const char * fmt, ...) {
     fflush(stdout);
 
-    fprintf(stderr, "%s:%d: ", file, line);
+    char message[2048];
+    int offset = snprintf(message, sizeof(message), "%s:%d: ", file, line);
 
     va_list args;
     va_start(args, fmt);
-    vfprintf(stderr, fmt, args);
+    vsnprintf(message + offset, sizeof(message) - offset, fmt, args);
     va_end(args);
 
-    fprintf(stderr, "\n");
+    if (g_abort_callback) {
+        g_abort_callback(message);
+    } else {
+        // default: print error and backtrace to stderr
+        fprintf(stderr, "%s\n", message);
+        ggml_print_backtrace();
+    }
 
-    ggml_print_backtrace();
     abort();
 }