Repository: AUTOMATIC1111/stable-diffusion-webui

Issue:
Uncapped token limit does not work past the ~85th token
**Describe the bug**
Tokens past ~85 do not really affect the prompt in a expected or consistant way. Tokens past ~90 do not do anything (except provide noise). It would be nice if someone could try on NovelAI (as they have a cap of ~200) if the behavior is the same.

**To Reproduce**
I have generated an image with the same seed multiple times, with the text `green eyes` at different token positions and varying attention. Commas are used as an unobtrusive padding to help get up to the nth token, without changing the image in any significant way. The end of the prompt is also padded up to the 99th token.

All images are generated with: Steps: 20, Sampler: Euler, CFG scale: 11, Seed: 2347389309, Size: 512x640, Model hash: 925997e9, Clip skip: 2
Negative prompt: `lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, artist name`
Using a certain trendy model (final-pruned), with the VAE and no hypernetworks. This model has been demonstrated to support up to ~200 tokens on the premium UI.

With `green eyes` at the:

- **76**th: `masterpiece, 1girl, cute, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , green eyes, , , , , , , , , , , , , , , , , , , , , ,` (pass)
![86](https://user-images.githubusercontent.com/112723046/194773620-b109e954-60ee-4b9f-9bf3-34d4ee1ee699.png)
- **81**st: `masterpiece, 1girl, cute, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , green eyes , , , , , , , , , , , , , , , , ,` (fail)
![81](https://user-images.githubusercontent.com/112723046/194773920-e710777e-03f3-4bad-a41c-75a6a6eba55d.png)
- Still **81**st (x1.5 attn): `masterpiece, 1girl, cute, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , (green eyes:1.5), , , , , , , , , , , , , , , , ,` (pass, attention helped)
![86_1 5](https://user-images.githubusercontent.com/112723046/194774011-955c4559-aa85-48f1-9c59-34cfb84fb4cd.png)
- **86**th (x1.5 attn): `masterpiece, 1girl, cute, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , (green eyes:1.5), , , , , , , , , , , ,` (fail, eyes are no longer green)
![86_1 5](https://user-images.githubusercontent.com/112723046/194774117-7d5e6f4b-d38c-406a-852c-daff7c8fe8d0.png)
- Still **86**th (x2.0 attn): `masterpiece, 1girl, cute, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , (green eyes:2.0), , , , , , , , , , , ,` (fail, increasing multiplier did not make the eyes green)
![86_2 0](https://user-images.githubusercontent.com/112723046/194774157-a00e10cf-28c1-471f-8f11-2fc9f384ae5c.png)
- Still **86**th (x2.5 attn): `masterpiece, 1girl, cute, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , (green eyes:2.5), , , , , , , , , , , ,` (fail, increasing any more would just destroy the image)
![86_2 5](https://user-images.githubusercontent.com/112723046/194774304-73fc0333-98a9-4515-9ac9-df195e4afbe6.png)
- **91**st (x2.5 attn): `masterpiece, 1girl, cute, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , (green eyes:2.5), , , , , , ,` (fail, no green at all now) 
![91_2 5](https://user-images.githubusercontent.com/112723046/194774411-4e509ae8-5325-40eb-932c-e606cb5d4eac.png)


**Additional context**
I've only demonstrated with 1 seed here, but the behavior is consistent with any seed.
Commas are used, but behavior is the same with normal words/tags.


Discussion:

Resolving patch:
diff --git a/modules/sd_hijack.py b/modules/sd_hijack.py
--- a/modules/sd_hijack.py
+++ b/modules/sd_hijack.py
@@ -43,10 +43,7 @@ def undo_optimizations():
 
 
 def get_target_prompt_token_count(token_count):
-    if token_count < 75:
-        return 75
-
-    return math.ceil(token_count / 10) * 10
+    return math.ceil(max(token_count, 1) / 75) * 75
 
 
 class StableDiffusionModelHijack:
@@ -127,7 +124,6 @@ def __init__(self, wrapped, hijack):
                 self.token_mults[ident] = mult
 
     def tokenize_line(self, line, used_custom_terms, hijack_comments):
-        id_start = self.wrapped.tokenizer.bos_token_id
         id_end = self.wrapped.tokenizer.eos_token_id
 
         if opts.enable_emphasis:
@@ -154,7 +150,13 @@ def tokenize_line(self, line, used_custom_terms, hijack_comments):
                     i += 1
                 else:
                     emb_len = int(embedding.vec.shape[0])
-                    fixes.append((len(remade_tokens), embedding))
+                    iteration = len(remade_tokens) // 75
+                    if (len(remade_tokens) + emb_len) // 75 != iteration:
+                        rem = (75 * (iteration + 1) - len(remade_tokens))
+                        remade_tokens += [id_end] * rem
+                        multipliers += [1.0] * rem
+                        iteration += 1
+                    fixes.append((iteration, (len(remade_tokens) % 75, embedding)))
                     remade_tokens += [0] * emb_len
                     multipliers += [weight] * emb_len
                     used_custom_terms.append((embedding.name, embedding.checksum()))
@@ -162,10 +164,10 @@ def tokenize_line(self, line, used_custom_terms, hijack_comments):
 
         token_count = len(remade_tokens)
         prompt_target_length = get_target_prompt_token_count(token_count)
-        tokens_to_add = prompt_target_length - len(remade_tokens) + 1
+        tokens_to_add = prompt_target_length - len(remade_tokens)
 
-        remade_tokens = [id_start] + remade_tokens + [id_end] * tokens_to_add
-        multipliers = [1.0] + multipliers + [1.0] * tokens_to_add
+        remade_tokens = remade_tokens + [id_end] * tokens_to_add
+        multipliers = multipliers + [1.0] * tokens_to_add
 
         return remade_tokens, fixes, multipliers, token_count
 
@@ -260,29 +262,55 @@ def process_text_old(self, text):
             hijack_fixes.append(fixes)
             batch_multipliers.append(multipliers)
         return batch_multipliers, remade_batch_tokens, used_custom_terms, hijack_comments, hijack_fixes, token_count
-
+    
     def forward(self, text):
-
-        if opts.use_old_emphasis_implementation:
+        use_old = opts.use_old_emphasis_implementation
+        if use_old:
             batch_multipliers, remade_batch_tokens, used_custom_terms, hijack_comments, hijack_fixes, token_count = self.process_text_old(text)
         else:
             batch_multipliers, remade_batch_tokens, used_custom_terms, hijack_comments, hijack_fixes, token_count = self.process_text(text)
 
-        self.hijack.fixes = hijack_fixes
         self.hijack.comments += hijack_comments
 
         if len(used_custom_terms) > 0:
             self.hijack.comments.append("Used embeddings: " + ", ".join([f'{word} [{checksum}]' for word, checksum in used_custom_terms]))
+        
+        if use_old:
+            self.hijack.fixes = hijack_fixes
+            return self.process_tokens(remade_batch_tokens, batch_multipliers)
+        
+        z = None
+        i = 0
+        while max(map(len, remade_batch_tokens)) != 0:
+            rem_tokens = [x[75:] for x in remade_batch_tokens]
+            rem_multipliers = [x[75:] for x in batch_multipliers]
+            
+            self.hijack.fixes = []
+            for unfiltered in hijack_fixes:
+                fixes = []
+                for fix in unfiltered:
+                    if fix[0] == i:
+                        fixes.append(fix[1])
+                self.hijack.fixes.append(fixes)
+            
+            z1 = self.process_tokens([x[:75] for x in remade_batch_tokens], [x[:75] for x in batch_multipliers])
+            z = z1 if z is None else torch.cat((z, z1), axis=-2)
+            
+            remade_batch_tokens = rem_tokens
+            batch_multipliers = rem_multipliers
+            i += 1
+        
+        return z
+        
+    
+    def process_tokens(self, remade_batch_tokens, batch_multipliers):
+        if not opts.use_old_emphasis_implementation:
+            remade_batch_tokens = [[self.wrapped.tokenizer.bos_token_id] + x[:75] + [self.wrapped.tokenizer.eos_token_id] for x in remade_batch_tokens]
+            batch_multipliers = [[1.0] + x[:75] + [1.0] for x in batch_multipliers]
+        
+        tokens = torch.asarray(remade_batch_tokens).to(device)
+        outputs = self.wrapped.transformer(input_ids=tokens, output_hidden_states=-opts.CLIP_stop_at_last_layers)
 
-        target_token_count = get_target_prompt_token_count(token_count) + 2
-
-        position_ids_array = [min(x, 75) for x in range(target_token_count-1)] + [76]
-        position_ids = torch.asarray(position_ids_array, device=devices.device).expand((1, -1))
-
-        remade_batch_tokens_of_same_length = [x + [self.wrapped.tokenizer.eos_token_id] * (target_token_count - len(x)) for x in remade_batch_tokens]
-        tokens = torch.asarray(remade_batch_tokens_of_same_length).to(device)
-
-        outputs = self.wrapped.transformer(input_ids=tokens, position_ids=position_ids, output_hidden_states=-opts.CLIP_stop_at_last_layers)
         if opts.CLIP_stop_at_last_layers > 1:
             z = outputs.hidden_states[-opts.CLIP_stop_at_last_layers]
             z = self.wrapped.transformer.text_model.final_layer_norm(z)
@@ -290,7 +318,7 @@ def forward(self, text):
             z = outputs.last_hidden_state
 
         # restoring original mean is likely not correct, but it seems to work well to prevent artifacts that happen otherwise
-        batch_multipliers_of_same_length = [x + [1.0] * (target_token_count - len(x)) for x in batch_multipliers]
+        batch_multipliers_of_same_length = [x + [1.0] * (75 - len(x)) for x in batch_multipliers]
         batch_multipliers = torch.asarray(batch_multipliers_of_same_length).to(device)
         original_mean = z.mean()
         z *= batch_multipliers.reshape(batch_multipliers.shape + (1,)).expand(z.shape)


Test patch: