anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: stanford-crfm/helm
Issue: stop sequences don't seem to work

When querying GPT-3, stop sequences doesn't seem to be respected.

http://crfm-models.stanford.edu/static/index.html?prompt=Answer%20the%20following%20question%20about%20geography.%0A%0AQuestion%3A%20What%20is%20the%20longest%20river%3F%0AAnswer%3A%20Nile%20%23%23%0A%0AQuestion%3A%20What%20is%20the%20tallest%20mountain%3F%0AAnswer%3A&settings=temperature%3A%200%0Astop_sequences%3A%20%5B%22%23%23%22%5D%0Atop_k_per_token%3A%205&environments=

Changed files:
src/proxy/openai_client.py

diff --git a/src/proxy/openai_client.py b/src/proxy/openai_client.py
index d6033a18..daeed70b 100644
--- a/src/proxy/openai_client.py
+++ b/src/proxy/openai_client.py
@@ -45,6 +45,13 @@ class OpenAIClient(Client):
             for text, logprob, top_logprobs in zip(
                 raw_data["tokens"], raw_data["token_logprobs"], raw_data["top_logprobs"]
             ):
+                # Do not include these excess tokens in the response.
+                # TODO: this is a hacky solution until we figure out why
+                #       OpenAI is sending tokens including and past the stop sequences.
+                # TODO: This logic doesn't work when the stop sequences spans multiple tokens.
+                if text.strip() in request.stop_sequences:
+                    break
+
                 # For some reason, the first log probability and top choices are None.
                 # TODO: look in to why this is
                 tokens.append(Token(text=text, logprob=logprob or 0, top_logprobs=dict(top_logprobs or {})))