Repository: ggml-org/llama.cpp
Pull request: fix: prevent segfault in tokenizer on highly repetitive input

Add nosubs|optimize flags to std::regex constructors to prevent catastrophic backtracking when processing prompts with repeated identical characters (e.g., 'A' * 10000).

The nosubs flag disables subgroup capture, significantly reducing memory usage and backtracking on uniform token sequences

*Make sure to read the [contributing guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md) before submitting a PR*

Before :
```
/root/llama.cpp.pascal/build/bin/llama-server --port 8088 -m /var/www/ia/models/lmstudio-community/gpt-oss-20b-GGUF/gpt-oss-20b-MXFP4.gguf

You are a helpful assistant

user
Hello

assistant
final
Hi there

user
How are you?

assistant'
main: model loaded
main: server is listening on http://127.0.0.1:8088
main: starting the main loop...
srv  update_slots: all slots are idle
Erreur de segmentation
```
(segfault)

After :
```
(root|~/llama.cpp.pascal) curl -X POST http://localhost:8088/v1/chat/completions   -H "Content-Type: application/json"   -d '{"messages":[{"role":"user","content":"'"$(python3 -c "print('A'*10000)")"' Say OK"}]}'
{"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","reasoning_content":"The user typed a long string of \"A\" and then says \"Say OK\". So they likely want the assistant to respond with \"OK\". The instruction: \"Say OK\". So we just reply \"OK\". But also must obey system instruction: We should not mention policy. Just reply \"OK\".","content":"OK"}}],"created":1764928092,"model":"gpt-oss-20b-MXFP4.gguf","system_fingerprint":"b7321-147310d71","object":"chat.completion","usage":{"completion_tokens":72,"prompt_tokens":1319,"total_tokens":1391},"id":"chatcmpl-i1GMuuGvb2X3irH73aTbLP0ZkwBYbJdf","timings":{"cache_n":0,"prompt_n":1319,"prompt_ms":171.43,"prompt_per_token_ms":0.12996967399545112,"prompt_per_second":7694.102549145423,"predicted_n":72,"predicted_ms":199.156,"predicted_per_token_ms":2.7660555555555555,"predicted_per_second":361.52563819317515}}
```

Close #17636


Changed files:
filename: src/unicode.cpp
patch: @@ -499,7 +499,7 @@ static std::vector<size_t> unicode_regex_split_custom_llama3(const std::string &
 
 // use std::wregex to split the text
 static std::vector<size_t> unicode_regex_split_stl(const std::wstring & wtext, const std::wstring & regex_expr, const std::vector<size_t> & offsets) {
-    std::wregex expr(regex_expr);
+    std::wregex expr(regex_expr, std::regex_constants::optimize | std::regex_constants::nosubs);
     std::vector<size_t> bpe_offsets; // store the offset of each word
     bpe_offsets.reserve(offsets.size()); // Reserve memory for the approximate size
     size_t start = 0;
@@ -529,7 +529,7 @@ static std::vector<size_t> unicode_regex_split_stl(const std::wstring & wtext, c
 
 // use std::regex to split the text
 static std::vector<size_t> unicode_regex_split_stl(const std::string & text, const std::string & regex_expr, const std::vector<size_t> & offsets) {
-    std::regex expr(regex_expr);
+    std::regex expr(regex_expr, std::regex_constants::optimize | std::regex_constants::nosubs);
     std::vector<size_t> bpe_offsets; // store the offset of each word
     bpe_offsets.reserve(offsets.size()); // Reserve memory for the approximate size
     size_t start = 0;