Instructions to use RichardErkhov/abacusai_-_bigyi-15b-4bits with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RichardErkhov/abacusai_-_bigyi-15b-4bits with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RichardErkhov/abacusai_-_bigyi-15b-4bits")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RichardErkhov/abacusai_-_bigyi-15b-4bits")
model = AutoModelForCausalLM.from_pretrained("RichardErkhov/abacusai_-_bigyi-15b-4bits", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RichardErkhov/abacusai_-_bigyi-15b-4bits with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RichardErkhov/abacusai_-_bigyi-15b-4bits"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RichardErkhov/abacusai_-_bigyi-15b-4bits",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/RichardErkhov/abacusai_-_bigyi-15b-4bits

SGLang

How to use RichardErkhov/abacusai_-_bigyi-15b-4bits with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RichardErkhov/abacusai_-_bigyi-15b-4bits" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RichardErkhov/abacusai_-_bigyi-15b-4bits",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RichardErkhov/abacusai_-_bigyi-15b-4bits" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RichardErkhov/abacusai_-_bigyi-15b-4bits",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use RichardErkhov/abacusai_-_bigyi-15b-4bits with Docker Model Runner:
```
docker model run hf.co/RichardErkhov/abacusai_-_bigyi-15b-4bits
```

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Quantization made by Richard Erkhov.

Github

Discord

Request more models

bigyi-15b - bnb 4bits

Model creator: https://huggingface.co/abacusai/
Original model: https://huggingface.co/abacusai/bigyi-15b/

Original model description:

base_model: - 01-ai/Yi-9B library_name: transformers tags: - mergekit - merge license: other license_name: yi-license license_link: LICENSE

bigyi-15b

I recently made bigstral-12b and then I saw this new awesome model yi-9b and decided to make an embiggened version.

This is a merge of pre-trained language models created using mergekit.

Bigyi-15b is a base / completion model, so there is no chat template.

It has a 4k context.

Example

Here is a recipe for Mai Tai:\n\n1:

3 parts rum, 2: 3 parts pineapple juice, 3: half a cup of lime juice, 4: 6 to 8 fresh or frozen pineapple chunks, 5: crushed ice. Mix all ingredients except ice and pour into glasses with ice. Garnish with a pineapple slice.

Here is an implementation of 2-sum in golang:

func twoSum(nums []int, target int) []int {
  if len(nums) <= 1 { return nil }
  m := map[int] bool{}
  for i := range(nums) {{
    n = nums[i]

    // find the complement of current number in map
    comp = target - n
    if comp in m { return [m[comp], i+1 ] }
    else { m[n] = true }
  }}
  return nil
}

Merge Details

Merge Method

This model was merged using the passthrough merge method.

Models Merged

The following models were included in the merge:

01-ai/Yi-9B

Configuration

The following YAML configuration was used to produce this model:

dtype: float16
merge_method: passthrough
slices:
- sources:
  - layer_range: [0, 12]
    model: 01-ai/Yi-9B
- sources:
  - layer_range: [6, 18]
    model: 01-ai/Yi-9B
- sources:
  - layer_range: [12, 24]
    model: 01-ai/Yi-9B
- sources:
  - layer_range: [18, 30]
    model: 01-ai/Yi-9B
- sources:
  - layer_range: [24, 36]
    model: 01-ai/Yi-9B
- sources:
  - layer_range: [30, 42]
    model: 01-ai/Yi-9B
- sources:
  - layer_range: [36, 48]
    model: 01-ai/Yi-9B

Downloads last month: 6

Safetensors

Model size

15B params

Tensor type

F32

F16