Repository: xorbitsai/inference

Issue:
worker metrics is empty
In version 0.10.3, use the command xinference-worker -e http://xinference-supervisor:9997 --worker-port 30001 --metrics-exporter-host xinference-worker --metrics-exporter-port 30002 to start the worker , access http://xinference-worker:30002/metrics through the browser, the results are as follows:

![image](https://github.com/xorbitsai/inference/assets/23291905/38fef7db-d39d-45a7-bbc9-44faa9f92717)


no indicator values ​​are displayed. Please help guide me as to where the configuration problem is?




Discussion:
@amumu96 Could you help?
And @codingl2k1 could give some advice as well.
Your screen shot shows that the metrics works correctly. https://inference.readthedocs.io/en/latest/user_guide/metrics.html
![image](https://github.com/xorbitsai/inference/assets/138426806/205c8892-94ed-468b-a05a-ccb0cbbb5a50)
This is my test, currently the metrics work well for the pytorch backend and vLLM backend.
> ![图像](https://private-user-images.githubusercontent.com/138426806/325744293-205c8892-94ed-468b-a05a-ccb0cbbb5a50.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTQxMDgyNzksIm5iZiI6MTcxNDEwNzk3OSwicGF0aCI6Ii8xMzg0MjY4MDYvMzI1NzQ0MjkzLTIwNWM4ODkyLTk0ZWQtNDY4Yi1hMDVhLWNjYjBjYmJiNWE1MC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNDI2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDQyNlQwNTA2MTlaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05ODQ2MGRlYTQ1OGY1ODg5ZThiMWM4YjQ4ODgwMTI4OWYyYTIxZmYyZmU3MGI5OWExZThjM2U5ZmY3OTg4NWYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.2iFHJbNSTr_sipaWeNcr2vonns6UZHomOCLixH-sDFQ) 这是我的测试，目前这些指标对于 pytorch 后端和 vLLM 后端运行良好。

Then why are there no indicator values ​​displayed?
> > ![图像](https://private-user-images.githubusercontent.com/138426806/325744293-205c8892-94ed-468b-a05a-ccb0cbbb5a50.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTQxMDgyNzksIm5iZiI6MTcxNDEwNzk3OSwicGF0aCI6Ii8xMzg0MjY4MDYvMzI1NzQ0MjkzLTIwNWM4ODkyLTk0ZWQtNDY4Yi1hMDVhLWNjYjBjYmJiNWE1MC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNDI2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDQyNlQwNTA2MTlaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05ODQ2MGRlYTQ1OGY1ODg5ZThiMWM4YjQ4ODgwMTI4OWYyYTIxZmYyZmU3MGI5OWExZThjM2U5ZmY3OTg4NWYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.2iFHJbNSTr_sipaWeNcr2vonns6UZHomOCLixH-sDFQ) 这是我的测试，目前这些指标对于 pytorch 后端和 vLLM 后端运行良好。
> 
> Then why are there no indicator values ​​displayed?

Which method did you use to call the model for chatting？ `/v1/chat/completions`？

Resolving patch:
diff --git a/xinference/core/model.py b/xinference/core/model.py
--- a/xinference/core/model.py
+++ b/xinference/core/model.py
@@ -25,6 +25,7 @@
     AsyncGenerator,
     Callable,
     Dict,
+    Generator,
     Iterator,
     List,
     Optional,
@@ -379,8 +380,13 @@ async def chat(self, prompt: str, *args, **kwargs):
             raise AttributeError(f"Model {self._model.model_spec} is not for chat.")
         finally:
             # For the non stream result.
-            if response is not None and isinstance(response, dict):
-                usage = response["usage"]
+            record = None
+            if isinstance(response, Generator) or isinstance(response, AsyncGenerator):
+                record = response
+            elif isinstance(response, bytes):
+                record = json.loads(response)
+            if record and isinstance(record, dict):
+                usage = record["usage"]
                 # Some backends may not have a valid usage, we just skip them.
                 completion_tokens = usage["completion_tokens"]
                 prompt_tokens = usage["prompt_tokens"]


Test patch:
diff --git a/xinference/core/tests/test_metrics.py b/xinference/core/tests/test_metrics.py
--- a/xinference/core/tests/test_metrics.py
+++ b/xinference/core/tests/test_metrics.py
@@ -82,3 +82,25 @@ async def test_metrics_exporter_server(setup_cluster):
     response = requests.get(metrics_exporter_address)
     assert response.ok
     assert 'xinference:input_tokens_total_counter{model="orca"} 1' in response.text
+
+
+async def test_metrics_exporter_data(setup_cluster):
+    endpoint, metrics_exporter_address, supervisor_address = setup_cluster
+
+    from ...client import Client
+
+    client = Client(endpoint)
+
+    model_uid = client.launch_model(
+        model_name="orca",
+        model_size_in_billions=3,
+        model_format="ggmlv3",
+        quantization="q4_0",
+    )
+
+    model = client.get_model(model_uid)
+    response = model.chat("write a poem.")
+
+    response = requests.get(metrics_exporter_address)
+    assert response.ok
+    assert 'format="ggmlv3",model="orca"' in response.text