ZK0622/v0.1mistral_lora_sft

Model Card for Model ID

Model Details

Model Description

Mistral 7b finetuned via 2 stage fine tunning (DAPT & SFT) with Hierarchical Merging and PEFT method (LoRA) on legal documents for the purpose of Legal Docs Summarization

• Developed by: [ZK]
• Model type: [Decoder Only]
• Finetuned from model: [Mistral 7b]

Model Sources

• Repository: [mistralai/Mistral-7B-v0.3]
• Paper: [https://arxiv.org/pdf/2310.06825]

Training Details

Contact owner for training details.

Training Data

[Only Data for SFT available "ZK0622/SFT"]

Evaluation

Testing Data, Factors & Metrics

Testing Data

[TO-BE-UPDATED]

Factors

[TO-BE-UPDATED]

Metrics

[TO-BE-UPDATED]

Results

PERPLEXITY

Model	Perplexity (Lower Better)
Mistral 7B Instruct v0.2	4.29
AdaLoRA Mistral 7B	3.73
LoRA Mistral 7B	(This Model) 3.54

ROUGE

Transformer Architecture	Model Name	Rouge-1	Rouge-2	Rouge-L	Rouge-L Sum
Encoder	Legal BERT	0.374	0.185	0.218	0.218
Encoder-Decoder	Legal LED	0.306	0.113	0.169	0.169
Encoder-Decoder	Legal T5	0.268	0.084	0.167	0.167
Decoder	Mistral 7B Base	0.440	0.195	0.269	0.273
Decoder	AdaLoRA Mistral 7B	0.506	0.294	0.350	0.351
Decoder (This Model)	LoRA Mistral 7B	0.564	0.376	0.437	0.437

Transformer Architecture	Model Name	Precision	Recall	BERT-F1
Encoder	Legal BERT	0.800	0.857	0.827
Encoder-Decoder	Legal LED	0.798	0.845	0.821
Encoder-Decoder	Legal T5	0.674	0.820	0.739
Decoder	Mistral 7B Base	0.852	0.866	0.858
Decoder	AdaLoRA Mistral 7B	0.874	0.884	0.878
Decoder (This Model)	LoRA Mistral 7B	0.908	0.890	0.898

Summary

Framework versions

• PEFT 0.10.0
• Transformer 4.40
• DeepSpeed config: ZeRO Stage 2