Edit

README.md

@@ -1,3 +1,33 @@
+# dinov2_vitl14_trt_a4000_fp16
+
+
+## Triton
+
+```
+make triton
+```
+
+## Build TensorRT Model
+
+```
+make model
+```
+
+
+```
+make trt
+```
+
+```
+tree model_repository
+```
+```
+model_repository/
+└── dinov2_vitl14
+    ├── 1
+    │   └── model.plan
+    └── config.pbtxt
+```
 
 
 ## Perf
@@ -7,7 +37,6 @@ make perf
 ```
 
 ```
-make perf
 docker run --gpus all --rm -it --net host nvcr.io/nvidia/tritonserver:23.04-py3-sdk perf_analyzer -m dinov2_vitl14 --percentile=95 -i grpc -u 0.0.0.0:6001 --concurrency-range 16:16 --shape input:3,560,560
 
 =================================
@@ -40,19 +69,19 @@ NOTE: CUDA Forward Compatibility mode ENABLED.
 
 Request concurrency: 16
   Client:
-    Request count: 1124
-    Throughput: 62.4339 infer/sec
-    p50 latency: 257390 usec
-    p90 latency: 287307 usec
-    p95 latency: 295432 usec
-    p99 latency: 305031 usec
-    Avg gRPC time: 254273 usec ((un)marshal request/response 801 usec + response wait 253472 usec)
+    Request count: 4009
+    Throughput: 222.66 infer/sec
+    p50 latency: 70762 usec
+    p90 latency: 83940 usec
+    p95 latency: 90235 usec
+    p99 latency: 102226 usec
+    Avg gRPC time: 71655 usec ((un)marshal request/response 741 usec + response wait 70914 usec)
   Server:
-    Inference count: 1124
-    Execution count: 202
-    Successful request count: 1124
-    Avg request latency: 248791 usec (overhead 9381 usec + queue 68460 usec + compute input 39 usec + compute infer 94051 usec + compute output 76859 usec)
+    Inference count: 4009
+    Execution count: 728
+    Successful request count: 4009
+    Avg request latency: 66080 usec (overhead 8949 usec + queue 16114 usec + compute input 1163 usec + compute infer 24751 usec + compute output 15103 usec)
 
 Inferences/Second vs. Client p95 Batch Latency
-Concurrency: 16, throughput: 62.4339 infer/sec, latency 295432 usec
+Concurrency: 16, throughput: 222.66 infer/sec, latency 90235 usec
 ```