Repository: guidance-ai/llguidance (Rust)

Issue:
Ignored terminal is reject on the first token
I am trying to use LLGuidance to generate Java program, but I found LLGuidance might reject the first token mysteriously. After debugging, I get the following reproduction code.

In this code, we only accept 'whitespaces', and C-style block comments are ignored. When the first character is part of the ignored block comment, LLGuidance rejects it. If a space is added before the first token, LLGuidance can accept the text normally.

I suspect this issue is related to #123.

I am using LLGuidance 1.2.0 and Qwen 2.5.

```Python
import transformers
import llguidance
import llguidance.hf
import llguidance.torch
import torch

grammar = r'''%llguidance { "no_forcing": true }
start: WS*
%import common.WS
IGNORED: /\/\*.*?\*\//s
%ignore IGNORED
'''

tokenizer = transformers.AutoTokenizer.from_pretrained('Qwen/Qwen2.5-0.5B')
llg_tokenizer = llguidance.hf.from_tokenizer(tokenizer)
mask_data = llguidance.torch.allocate_token_bitmask(1, llg_tokenizer.vocab_size)
interp = llguidance.LLMatcher(llg_tokenizer, grammar)
assert not interp.is_error(), interp.get_error()

def process_logit(input_ids: list[int], logit: torch.Tensor) -> torch.Tensor:
  if input_ids:
    last_token = input_ids[-1]
    assert mask_data[0, last_token // 32] & (1 << (last_token % 32)) != 0
    interp.consume_token(last_token)
    assert not interp.is_error(), interp.get_error()
  llguidance.torch.fill_next_token_bitmask(interp, mask_data)
  llguidance.torch.apply_token_bitmask_inplace(logit, mask_data)
  return logit

def process_tokens(tokens: list[int]):
  for i in range(len(tokens)):
    logit = torch.zeros((llg_tokenizer.vocab_size,), dtype=torch.float)
    visible_tokens = tokens[:i]
    masked_logit = process_logit(visible_tokens, logit)
    assert masked_logit[tokens[i]] != float('-inf'), f'token {i} ({tokens[i]}, {tokenizer.decode(tokens[i])!r}) is masked'

text = '/*  */'
tokens = tokenizer.encode(text)
print(tokens)
process_tokens(tokens)
```


Discussion:
Actually, it seems like `llguidance` will simply reject a token that can only consistude an ignored terminal at the beginning of text. The example does not need to use an ignored terminal defined by a regex.
An updated version of the reproduction code would be this.
```Python
import transformers
import llguidance
import llguidance.hf
import llguidance.torch
import torch

grammar = r'''%llguidance { "no_forcing": true }
start: "a"*
IGNORED: "b"
%ignore IGNORED
'''

tokenizer = transformers.AutoTokenizer.from_pretrained('Qwen/Qwen2.5-0.5B')
llg_tokenizer = llguidance.hf.from_tokenizer(tokenizer)
mask_data = llguidance.torch.allocate_token_bitmask(1, llg_tokenizer.vocab_size)
interp = llguidance.LLMatcher(llg_tokenizer, grammar)
assert not interp.is_error(), interp.get_error()

def process_logit(input_ids: list[int], logit: torch.Tensor) -> torch.Tensor:
  if input_ids:
    last_token = input_ids[-1]
    assert mask_data[0, last_token // 32] & (1 << (last_token % 32)) != 0
    interp.consume_token(last_token)
    assert not interp.is_error(), interp.get_error()
  llguidance.torch.fill_next_token_bitmask(interp, mask_data)
  llguidance.torch.apply_token_bitmask_inplace(logit, mask_data)
  return logit

def process_tokens(tokens: list[int]):
  for i in range(len(tokens)):
    logit = torch.zeros((llg_tokenizer.vocab_size,), dtype=torch.float)
    visible_tokens = tokens[:i]
    masked_logit = process_logit(visible_tokens, logit)
    assert masked_logit[tokens[i]] != float('-inf'), f'token {i} ({tokens[i]}, {tokenizer.decode(tokens[i])!r}) is masked'

text = 'bbabbbaaa'
tokens = tokenizer.encode(text)
print(tokens)
process_tokens(tokens)
```
Hi @Gompyn -- we currently disallow the %ignore lexeme from actually being ignored / skipped at the very beginning of the grammar. Experimentally, we've often found that allowing this to be skipped often results in the LLM producing endless whitespace instead of actually starting the grammar (when whitespace is being ignored). Just something to look out for.

Opening a PR to expose this as an option to the user though. Thanks for opening the issue!



Patch:
diff --git a/docs/syntax.md b/docs/syntax.md
index c8fd31e1..c728bb5d 100644
--- a/docs/syntax.md
+++ b/docs/syntax.md
@@ -318,7 +318,7 @@ MULT_NUM: %regex {
 
 Certain grammar options can be set by using `%llguidnace { ... }`,
 by passing it a JSON object with the options;
-see `LLGuidanceOptions` in [api.rs](../parser/src/api.rs#L24).
+see `LLGuidanceOptions` in [api.rs](../parser/src/api.rs#L36).
 Example: `%llguidance { "no_forcing": true }`.
 It can be specified multiple times, with the options being merged.
 
diff --git a/parser/src/api.rs b/parser/src/api.rs
index 83f9687d..72f42c52 100644
--- a/parser/src/api.rs
+++ b/parser/src/api.rs
@@ -47,6 +47,13 @@ pub struct LLGuidanceOptions {
     /// This is very unlikely what you need.
     #[serde(default)]
     pub allow_invalid_utf8: bool,
+
+    /// If set, the grammar will allow the %ignore lexeme at the start of the grammar.
+    /// Otherwise, it will only be allowed after the first non-ignored lexeme.
+    /// This option (like the other options here) will apply to the entire grammar,
+    /// including nested sub-grammars.
+    #[serde(default)]
+    pub allow_initial_skip: bool,
 }
 
 impl LLGuidanceOptions {
@@ -57,6 +64,9 @@ impl LLGuidanceOptions {
         if other.allow_invalid_utf8 {
             self.allow_invalid_utf8 = true;
         }
+        if other.allow_initial_skip {
+            self.allow_initial_skip = true;
+        }
     }
 }
 
diff --git a/parser/src/grammar_builder.rs b/parser/src/grammar_builder.rs
index ee22b1ff..4950c1de 100644
--- a/parser/src/grammar_builder.rs
+++ b/parser/src/grammar_builder.rs
@@ -233,6 +233,9 @@ impl GrammarBuilder {
         if options.no_forcing {
             self.regex.spec.no_forcing = true;
         }
+        if options.allow_initial_skip {
+            self.regex.spec.allow_initial_skip = true;
+        }
 
         // add root node
         self.curr_start_idx = self.new_node("start");


Test patch:
diff --git a/sample_parser/tests/test_ll.rs b/sample_parser/tests/test_ll.rs
index 2d093090..eabb1f00 100644
--- a/sample_parser/tests/test_ll.rs
+++ b/sample_parser/tests/test_ll.rs
@@ -203,6 +203,15 @@ fn test_ll_llg_options() {
         }),
         &["", "JSON‧{\"‧a‧_‧long‧_‧property‧_‧name‧\":‧ ‧5‧}"],
     );
+    check_lark_grammar(
+        r#"
+            %llguidance { "allow_initial_skip": true }
+            start: "a"*
+            IGNORED: "b"
+            %ignore IGNORED
+        "#,
+        &["", "bb‧a‧bb‧b‧aa‧a‧≺EOS≻"],
+    );
 }
 
 #[test]