Repository: deepset-ai/haystack
Pull request: fix: Tiktoken does not support Azure gpt-35-turbo
### Related Issues
 - Fixes #4738 4738

### Proposed Changes:
 Just force the tokenizer if the model is `gpt-35-turbo` basically.

### Notes for the reviewer
This is currently in open PR with Tiktoken, but it's been sat there for a while: https://github.com/openai/tiktoken/pull/72

### Checklist
- [X] I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- [X] I have updated the related issue with new insights and changes
- [ ] I added tests that demonstrate the correct behavior of the change
- [x] I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:`.
- [x] I documented my code
- [x] I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Problem statement:
Azure's gpt-35-turbo gets wrong max_token_limit
**Describe the bug**
Due to Tiktoken's lack of support for Azure's `gpt-35-turbo`, the open_ai_utils.py script returns the wrong value for the max token length.

**Error message**
Returned token length is default of 2049

**Expected behavior**
Token length should be 4096

Hints from issue discussion:

Commit message: Remove trailing whitespace

Patch to review:
diff --git a/haystack/utils/openai_utils.py b/haystack/utils/openai_utils.py
index 66fef9c5ce..289fb7dd04 100644
--- a/haystack/utils/openai_utils.py
+++ b/haystack/utils/openai_utils.py
@@ -99,6 +99,10 @@ def _openai_text_completion_tokenization_details(model_name: str):
     max_tokens_limit = 2049  # Based on this ref: https://platform.openai.com/docs/models/gpt-3
     model_tokenizer = MODEL_TO_ENCODING.get(model_name) if USE_TIKTOKEN else None
 
+    # covering the lack of support in Tiktoken. https://github.com/openai/tiktoken/pull/72
+    if model_name == "gpt-35-turbo" and USE_TIKTOKEN:
+        model_tokenizer = "cl100k_base"
+
     if model_tokenizer:
         # Based on OpenAI models page, 'davinci' considers have 2049 tokens,
         ## therefore, it is better to add `text-davinci` instead to the condition.
@@ -110,6 +114,10 @@ def _openai_text_completion_tokenization_details(model_name: str):
         elif model_name.startswith("gpt-3"):
             max_tokens_limit = 4096
             tokenizer_name = model_tokenizer
+        # covering the lack of support in Tiktoken. https://github.com/openai/tiktoken/pull/72
+        elif model_name.startswith("gpt-35"):
+            max_tokens_limit = 4096
+            tokenizer_name = model_tokenizer
         # Ref: https://platform.openai.com/docs/models/gpt-4
         elif model_name.startswith("gpt-4-32k"):
             max_tokens_limit = 32768  # tokens


Code review comments:
diff_hunk: @@ -110,6 +114,10 @@ def _openai_text_completion_tokenization_details(model_name: str):
         elif model_name.startswith("gpt-3"):
             max_tokens_limit = 4096
             tokenizer_name = model_tokenizer
+        # covering the lack of support in Tiktoken. https://github.com/openai/tiktoken/pull/72
+        elif model_name.startswith("gpt-35"):
+            max_tokens_limit = 4096
+            tokenizer_name = model_tokenizer
path: haystack/utils/openai_utils.py
text: @user1:
This elif-block will never be reached as it's superseded by `elif model_name.startswith("gpt-3")` in l. 114. I guess we don't need this block here?

@author:
Good shout, it should get caught by the previous, yes.  Let me remove it :)