Eponec - A grammar and programming tool for guiding LLMs

chumpro: <p>Hi, I am doing a project where I create parser that can be used to guide LLMs.</p>
<p>Here is a link if you like to check it out:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/chumpro/eponec">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/chumpro/eponec" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/bc63f9030c861778689a544ef8692e945fe618832a2b539a242a9f4e546d7fba/chumpro/eponec" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/chumpro/eponec" target="_blank" rel="noopener nofollow ugc">GitHub - chumpro/eponec: A grammar and programming tool for guiding LLMs</a></h3>

  <p>A grammar and programming tool for guiding LLMs. Contribute to chumpro/eponec development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I have two, possibly related, problems. I am using huggingface transformers and TinyLlama/TinyLlama_v1.1. I’m on a laptop with no GPU.</p>
<p>Problem 1: I want to have a dictionary of token_string → token_id, where the token_string has the tokenizers special characters like space and newline translated into the normal ’ ', ‘\n’. I’m doing regex matching against the token strings. Currently I am achiving this by using the following function:</p>
<pre><code>def fix_token( s ) :
	return s.replace( chr( 9601 ), ' ' ).replace( '&lt;0x0A&gt;', '\n' )
</code></pre>
<p>I’m thinking there is a better way to achieve this by using the tokenizer.</p>
<p>I’ve tried tried the following :</p>
<pre><code>s1 = tokenizer.convert_ids_to_tokens( [ 29504 ] )

print( 's1', s1 )

s2 = tokenizer.convert_tokens_to_string( s1 )

print( 's2', [ s2 ] )

s3 = tokenizer.decode( [ 29504 ] )

print( 's3', [ s3 ] )
</code></pre>
<p>which outputs:</p>
<pre><code>s1 ['▁terminate']
s2 ['terminate']
s3 ['terminate']
</code></pre>
<p>Strangely convert_tokens_to_string handles newline “correctly”. The output I am hoping for starts with an ordinary space like this:</p>
<pre><code>[' terminate']
</code></pre>
<p>Problem 2: The second problem is that ‘’ sometimes encodes to something that decodes to ' ', that is adding a succeeding space.<br>
Notice that I am decoding the output_ids of model.generate ( to use for regex pattern matching ) and then I encode it again so that<br>
the model get the expected tokenization of the string.</p>
<p>I have the following ( the text prompt in <span class="chcklst-box fa fa-square-o fa-fw"></span>'s and input_ids is the encoding ):</p>
<pre><code>[' &lt;|system|&gt;\n']
input_ids tensor([29871,   529, 29989,  5205, 29989, 29958,    13])

[' &lt;|system|&gt;\nYou are an AI assistant.']
input_ids tensor([29871,   529, 29989,  5205, 29989, 29958,    13,  3492,   526,   385,
	      319, 29902, 20255, 29889])

[' &lt;|system|&gt;\nYou are an AI assistant.&lt;/s&gt;']
input_ids tensor([29871,   529, 29989,  5205, 29989, 29958,    13,  3492,   526,   385,
	      319, 29902, 20255, 29889,     2])

[' &lt;|system|&gt;\nYou are an AI assistant.&lt;/s&gt;&lt;|user|&gt;\n']
input_ids tensor([29871,   529, 29989,  5205, 29989, 29958,    13,  3492,   526,   385,
	      319, 29902, 20255, 29889,     2,   529, 29989,  1792, 29989, 29958,
	       13])
&gt; hi

[' &lt;|system|&gt;\nYou are an AI assistant.&lt;/s&gt;&lt;|user|&gt;\nhi']
input_ids tensor([29871,   529, 29989,  5205, 29989, 29958,    13,  3492,   526,   385,
	      319, 29902, 20255, 29889,     2,   529, 29989,  1792, 29989, 29958,
	       13,  2918])

[' &lt;|system|&gt;\nYou are an AI assistant.&lt;/s&gt;&lt;|user|&gt;\nhi&lt;|assistant|&gt;\n']
input_ids tensor([29871,   529, 29989,  5205, 29989, 29958,    13,  3492,   526,   385,
	      319, 29902, 20255, 29889,     2,   529, 29989,  1792, 29989, 29958,
	       13,  2918, 29966, 29989,   465, 22137, 29989, 29958,    13])

[' &lt;|system|&gt;\nYou are an AI assistant.&lt;/s&gt; &lt;|user|&gt;\nhi&lt;|assistant|&gt;\ni']
input_ids tensor([29871,   529, 29989,  5205, 29989, 29958,    13,  3492,   526,   385,
	      319, 29902, 20255, 29889,     2, 29871,   529, 29989,  1792, 29989,
	    29958,    13,  2918, 29966, 29989,   465, 22137, 29989, 29958,    13,
	    29875])
</code></pre>
<p>Notice that ‘’ has token id 2. Up until the last printout the 2 is followed by 529, the beginning of the sequence ‘&lt;|user|&gt;\n’. Then all of a sudden at the last step, a 29871 is inserted between the 2 and the 529. The code that is running here is eponec.Parser.generate with an extra printout of input_ids. What I’m trying to do here is make an eponec.Parser that does chat correctly according to the chat template of the model. The extra space is causing problems for the parser, which relies on offset.</p>
<p>I have a feeling I’m not doing this the right way. My question is: What is happening and how to stop it from happening?</p>

chumpro: <p>I have a workaround for these problems now.</p>