thenlper michaelfeil commited on
Commit
ec7d9de
1 Parent(s): c6c1b92

Readme, add infinity deployment documentation (#21)

Browse files

- Update README.md (31e97fe8a364fd9c580cbe8e5432c263213afc60)
- Update README.md (13661f9e67466c5451f26990e0df3efa9090ae3e)


Co-authored-by: Michael <michaelfeil@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +10 -0
README.md CHANGED
@@ -5620,6 +5620,16 @@ scores = (embeddings[:2] @ embeddings[2:].T) * 100
5620
  print(scores.tolist())
5621
  ```
5622
 
 
 
 
 
 
 
 
 
 
 
5623
  ## Evaluation
5624
 
5625
  ### MTEB & C-MTEB
 
5620
  print(scores.tolist())
5621
  ```
5622
 
5623
+ ### infinity_emb
5624
+
5625
+ Usage via [infinity, MIT Licensed](https://github.com/michaelfeil/infinity).
5626
+ ```bash
5627
+ docker run \
5628
+ --gpus "0" -p "7997":"7997" \
5629
+ michaelf34/infinity:0.0.68-trt-onnx \
5630
+ v2 --model-id Alibaba-NLP/gte-Qwen2-1.5B-instruct --revision "refs/pr/20" --dtype bfloat16 --batch-size 16 --device cuda --engine torch --port 7997 --no-bettertransformer
5631
+ ```
5632
+
5633
  ## Evaluation
5634
 
5635
  ### MTEB & C-MTEB