Repository: traceloop/openllmetry
Issue category: Performance Issue

Issue:
🐛 Bug Report: Performance issues with opentelemetry-instrumentation-openai
### Which component is this bug for?

Anthropic Instrumentation

### 📜 Description

Hi! First, thank you for opentelemetry-instrumentation-openai. It makes my life easier :-)

But, it's computationally extensive, significantly slowing the service of my team. I'm raising a bug report as I couldn't find a better issue type from the ones suggested. Feel free to change this as needed.

Overall, the impact of this issue is reduced throughput. In our production environment removing the library increased our throughput under load 6 folds! The minimal reproducible project [here](https://github.com/Nagasaki45/OTEL-OpenAI-performance-investigation) shows more than 4x throughput increase when dropping the library, without any other change.

### 👟 Reproduction steps

This is a bit elaborated, so I created a [minimal reproducible example](https://github.com/Nagasaki45/OTEL-OpenAI-performance-investigation) for demonstrating the problem and for going through the investigation steps. The README of that repo explains the steps in details. If more info is needed I'm happy to provide it.

### 👍 Expected behavior

Adding opentelemetry-instrumentation-openai to a project should have a minimal impact on a system's performance.

### 👎 Actual Behavior with Screenshots

Adding opentelemetry-instrumentation-openai to a project reduces the throughput under load by ~4-6 times.

### 🤖 Python Version

3.12.0

### 📃 Provide any additional context for the Bug.

_No response_

### 👀 Have you spent some time to check if this bug has been raised before?

- [x] I checked and didn't find similar issue

### Are you willing to submit PR?

Yes I am willing to submit a PR!
🐛 Bug Report: Performance issues with opentelemetry-instrumentation-openai
### Which component is this bug for?

Anthropic Instrumentation

### 📜 Description

Hi! First, thank you for opentelemetry-instrumentation-openai. It makes my life easier :-)

But, it's computationally extensive, significantly slowing the service of my team. I'm raising a bug report as I couldn't find a better issue type from the ones suggested. Feel free to change this as needed.

Overall, the impact of this issue is reduced throughput. In our production environment removing the library increased our throughput under load 6 folds! The minimal reproducible project [here](https://github.com/Nagasaki45/OTEL-OpenAI-performance-investigation) shows more than 4x throughput increase when dropping the library, without any other change.

### 👟 Reproduction steps

This is a bit elaborated, so I created a [minimal reproducible example](https://github.com/Nagasaki45/OTEL-OpenAI-performance-investigation) for demonstrating the problem and for going through the investigation steps. The README of that repo explains the steps in details. If more info is needed I'm happy to provide it.

### 👍 Expected behavior

Adding opentelemetry-instrumentation-openai to a project should have a minimal impact on a system's performance.

### 👎 Actual Behavior with Screenshots

Adding opentelemetry-instrumentation-openai to a project reduces the throughput under load by ~4-6 times.

### 🤖 Python Version

3.12.0

### 📃 Provide any additional context for the Bug.

_No response_

### 👀 Have you spent some time to check if this bug has been raised before?

- [x] I checked and didn't find similar issue

### Are you willing to submit PR?

Yes I am willing to submit a PR!


Hints:
Thanks @Nagasaki45 for reporting! We'll investigate what's causing it. Since you're using streaming I have an assumption it's related to the token count enrichment which may affect latency (but some users find useful!). Can you try disabling it and see the effect? (Setting `enrich_token_usage` to false in the initialization).
Thanks for the prompt response @nirga! I'm using OTel automatic instrumentation (i.e. `opentelemetry-instrument my_app.py`). Is there a way to control this config with environment variables or flags to `opentelemetry-instrument` itself? I tried to look it up on the docs / repo and didn't find an obvious answer.
I actually see it's off by default @Nagasaki45 so nvm. We'll take a look at it and keep you posted! This should have a negligible impact on performance. 
Thanks again @nirga. I might have the capacity to help investigate this. I now have decent understanding of how to measure this bottleneck so might be able to help. Let me know.
I looked a bit more into it and I think the issue is the `version` calls. There's one [here](https://github.com/traceloop/openllmetry/blob/3539e34026a06d4cccdc1b73fd26d3f6b97fee02/packages/opentelemetry-instrumentation-openai/opentelemetry/instrumentation/openai/utils.py#L14) and another one [here](https://github.com/traceloop/openllmetry/blob/3539e34026a06d4cccdc1b73fd26d3f6b97fee02/packages/opentelemetry-instrumentation-openai/opentelemetry/instrumentation/openai/shared/__init__.py#L238). I just tried to replace these with `True` / `False` matching the packages versions I have installed and I think the throughput is back where it should be. This is inconclusive because I'm testing on a different machine, so CPU load behaves a bit differently. I will test on the same machine I used to create the ticket and report back.
Interesting @Nagasaki45, that actually makes sense! We should cache them during initialization
Thanks @Nagasaki45 for reporting! We'll investigate what's causing it. Since you're using streaming I have an assumption it's related to the token count enrichment which may affect latency (but some users find useful!). Can you try disabling it and see the effect? (Setting `enrich_token_usage` to false in the initialization).
Thanks for the prompt response @nirga! I'm using OTel automatic instrumentation (i.e. `opentelemetry-instrument my_app.py`). Is there a way to control this config with environment variables or flags to `opentelemetry-instrument` itself? I tried to look it up on the docs / repo and didn't find an obvious answer.
I actually see it's off by default @Nagasaki45 so nvm. We'll take a look at it and keep you posted! This should have a negligible impact on performance. 
Thanks again @nirga. I might have the capacity to help investigate this. I now have decent understanding of how to measure this bottleneck so might be able to help. Let me know.
I looked a bit more into it and I think the issue is the `version` calls. There's one [here](https://github.com/traceloop/openllmetry/blob/3539e34026a06d4cccdc1b73fd26d3f6b97fee02/packages/opentelemetry-instrumentation-openai/opentelemetry/instrumentation/openai/utils.py#L14) and another one [here](https://github.com/traceloop/openllmetry/blob/3539e34026a06d4cccdc1b73fd26d3f6b97fee02/packages/opentelemetry-instrumentation-openai/opentelemetry/instrumentation/openai/shared/__init__.py#L238). I just tried to replace these with `True` / `False` matching the packages versions I have installed and I think the throughput is back where it should be. This is inconclusive because I'm testing on a different machine, so CPU load behaves a bit differently. I will test on the same machine I used to create the ticket and report back.
Interesting @Nagasaki45, that actually makes sense! We should cache them during initialization

Patch:
diff --git a/packages/opentelemetry-instrumentation-groq/opentelemetry/instrumentation/groq/utils.py b/packages/opentelemetry-instrumentation-groq/opentelemetry/instrumentation/groq/utils.py
index e283307b9..f8d750d4c 100644
--- a/packages/opentelemetry-instrumentation-groq/opentelemetry/instrumentation/groq/utils.py
+++ b/packages/opentelemetry-instrumentation-groq/opentelemetry/instrumentation/groq/utils.py
@@ -9,6 +9,8 @@
 GEN_AI_SYSTEM = "gen_ai.system"
 GEN_AI_SYSTEM_GROQ = "groq"
 
+_PYDANTIC_VERSION = version("pydantic")
+
 
 def set_span_attribute(span, name, value):
     if value is not None and value != "":
@@ -68,7 +70,7 @@ def error_metrics_attributes(exception):
 
 
 def model_as_dict(model):
-    if version("pydantic") < "2.0.0":
+    if _PYDANTIC_VERSION < "2.0.0":
         return model.dict()
     if hasattr(model, "model_dump"):
         return model.model_dump()
diff --git a/packages/opentelemetry-instrumentation-openai/opentelemetry/instrumentation/openai/shared/__init__.py b/packages/opentelemetry-instrumentation-openai/opentelemetry/instrumentation/openai/shared/__init__.py
index e54902ac5..87965a3fe 100644
--- a/packages/opentelemetry-instrumentation-openai/opentelemetry/instrumentation/openai/shared/__init__.py
+++ b/packages/opentelemetry-instrumentation-openai/opentelemetry/instrumentation/openai/shared/__init__.py
@@ -23,6 +23,8 @@
 PROMPT_FILTER_KEY = "prompt_filter_results"
 PROMPT_ERROR = "prompt_error"
 
+_PYDANTIC_VERSION = version("pydantic")
+
 # tiktoken encodings map for different model, key is model_name, value is tiktoken encoding
 tiktoken_encodings = {}
 
@@ -236,7 +238,7 @@ def is_streaming_response(response):
 def model_as_dict(model):
     if isinstance(model, dict):
         return model
-    if version("pydantic") < "2.0.0":
+    if _PYDANTIC_VERSION < "2.0.0":
         return model.dict()
     if hasattr(model, "model_dump"):
         return model.model_dump()
diff --git a/packages/opentelemetry-instrumentation-openai/opentelemetry/instrumentation/openai/utils.py b/packages/opentelemetry-instrumentation-openai/opentelemetry/instrumentation/openai/utils.py
index 9030e3004..e0ab375a1 100644
--- a/packages/opentelemetry-instrumentation-openai/opentelemetry/instrumentation/openai/utils.py
+++ b/packages/opentelemetry-instrumentation-openai/opentelemetry/instrumentation/openai/utils.py
@@ -9,9 +9,11 @@
 import openai
 from opentelemetry.instrumentation.openai.shared.config import Config
 
+_OPENAI_VERSION = version("openai")
+
 
 def is_openai_v1():
-    return version("openai") >= "1.0.0"
+    return _OPENAI_VERSION >= "1.0.0"
 
 
 def is_azure_openai(instance):


Test patch: