Wizard-Vicuna-13B-Uncensored-SpQR

Overview

This model is an SpQR 4-bit quantization of the original Wizard-Vicuna-13B-Uncensored-HF

Quantization Specifications

• Quantization: 4-bit, group size of 16, per-channel with scale and zero-point of 3 bits.
• Outliers: Threshold set at 0.2.
• Permutation Order: act_order.
• Dampening: Set at 1e0.
• Sampling: 128 samples.
• Logging: Via Weights & Biases.

Evaluation Metrics

The following perplexity scores were obtained on various datasets:

Dataset	Perplexity
c4	7.354
wikitext2	5.685
ptb	20.822