Edit model card

Description:

This is a llama 13b 4 bit 128g model quantized after merging the LoRA of the same name.

Objective for this project:

To create a model that upholds a logical thread, regardless of whether the output is verbose or concise. Training has been performed on a version of the pile of sets, reduced to 40% of its original size, to expedite training iterations. I personally utilize this model as an aid for storytelling and writing. While it serves this purpose adequately, I still perceive this version as a prototype.

Prompt format:

Stanford Alpaca

The prompt should start on a new line after "### Response:"

  • For examples with a non-empty input field:
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input}

### Response:
  • For examples with an empty input field:
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:

Perplexity Benchmarks:

  • wikitext: 4.66796875

Training information:

  • 2 Epochs
  • 64 / 32 R / A
  • 1024 Cutoff
  • 19 hours on an A6000

Data used in training:

All cleaned and scrubbed in various ways then culled to various degrees.

  • Camel biology, physics, chemistry, math, and AI society
  • Alpaca evol instruct
  • GPTeacher Instruct
  • Alpaca GPT4
  • Dolly Databricks

Plans for the future, a brief overview:

  • Pivot to a conversational format going forward
  • Train another 13b LoRA against the entirety of my pile of sets rather than just a portion of it for Mk2
  • Train 30b on the Mk2 pile of sets
  • Expand the story generation capabilities and likely more for Mk3

Model used for training and other information:

https://huggingface.co/PocketDoc/llama-13b-gptq-4bit-128g

CUDA_VISIBLE_DEVICES=0 python llama.py ~/text-generation-webui/models/Dans-PileOfSets-Mk1-llama-13b-merged c4 --wbits 4 --true-sequential --groupsize 128 --save_safetensors Dans-PileOfSets-Mk1-llama-13b-gptq-4bit-128g.safetensors

https://huggingface.co/PocketDoc/Dans-PileOfSets-Mk1-llama-13b-merged

Disclaimer:

It has not been aligned and no warranty is given for the quality or safety of its outputs.

Downloads last month
1