Model: aritrasen/bge-base-en-v1.5-ft
Author: aritrasen
Pipeline: sentence-similarity
Library: sentence-transformers
Tags: sentence-transformers, safetensors, bert, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:21, loss:MultipleNegativesRankingLoss, arxiv:1908.10084, arxiv:1705.00652, base_model:BAAI/bge-base-en-v1.5, base_model:finetune:BAAI/bge-base-en-v1.5, text-embeddings-inference, endpoints_compatible, region:us
Base models (finetune): id: BAAI/bge-base-en-v1.5
Card metadata: base_model: BAAI/bge-base-en-v1.5
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags: sentence-transformers, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:21, loss:MultipleNegativesRankingLoss
widget: source_sentence: | Config                            | Model                  | Epochs | Max seq length | Micro batch size | Machine | Training runtime | Cost | Peak memory | Validation loss | Validation perplexity | Multitask score (MMLU) |
| --------------------------------- | ---------------------- | ------ | -------------- | ---------------- | ------- | ---------------- | ---- | ----------- | --------------- | --------------------- | --------------- |
| falcon-7b/lora.yaml               | falcon-7b              | 4      | 512            | 1                | 1xA10G  | 24.84 min        | $0.7 | 16.69 GB    | 0.945           | 2.573                 | 26.2%           |
| falcon-7b/lora.yaml               | falcon-7b              | 4      | 512            | 1                | 4xA10G  | 24.94 min        | $2.0 | 16.69 GB    | 0.945           | 2.573                 | 26.4%           |
| falcon-7b/qlora.yaml              | falcon-7b              | 4      | 512            | 1                | 1xA10G  | 50.85 min        | $1.5 | 9.44 GB     | 0.993           | 2.699                 | 26.3%           |
| falcon-7b/qlora.yaml              | falcon-7b              | 4      | 512            | 1                | 4xA10G  | 50.88 min        | $4.1 | 9.44 GB     | 0.993           | 2.699                 | 26.3%           |
|                                   |                        |        |                |                  |         |                  |      |             |                 |                       |                 |
| gemma-2b/full.yaml                | gemma-2b               | 1      | 512            | 1                | 4xA10G  | 14.06 min        | $1.1 | 17.43 GB    | 1.021           | 2.777                 | 32.4%           |
| gemma-2b/lora.yaml                | gemma-2b               | 2      | 512            | 2                | 1xA10G  | 9.41 min         | $0.3 | 12.62 GB    | 0.981           | 2.666                 | 34.4%           |
sentences: What is the command to download the pretrained model weights for the Llama-2-7b-hf model?
, What is the version of nvfuser\_cu121 used?
, What is the training runtime for the gemma-2b model with the lora configuration?
, source_sentence: # Serve and Deploy LLMs

This document shows how you can serve a LitGPT for deployment. 

&nbsp;
## Serve an LLM

This section illustrates how we can set up an inference server for a phi-2 LLM using `litgpt serve` that is minimal and highly scalable.


&nbsp;
## Step 1: Start the inference server


```bash
# 1) Download a pretrained model (alternatively, use your own finetuned model)
litgpt download --repo_id microsoft/phi-2

# 2) Start the server
litgpt serve --checkpoint_dir checkpoints/microsoft/phi-2
```

> [!TIP]
> Use `litgpt serve --help` to display additional options, including the port, devices, LLM temperature setting, and more.


&nbsp;
## Step 2: Query the inference server

You can now send requests to the inference server you started in step 2. For example, in a new Python session, we can send requests to the inference server as follows:


```python
import requests, json

response = requests.post(
    "http://127.0.0.1:8000/predict", 
    json={"prompt": "Fix typos in the following sentence: Exampel input"}
)

print(response.json()["output"])
```

Executing the code above prints the following output:

```
Instruct: Fix typos in the following sentence: Exampel input
Output: Example input.
```
sentences: What command do I use to convert the finetuned model to a HF transformer model?
, How do you merge LoRA weights into the original model's checkpoint?
, How can I start an inference server for a phi-2 LLM using litgpt serve?

Likes: 0 | All-time downloads: 5
Created: 2024-07-17 14:12:45