File size: 3,121 Bytes
efb3463 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 |
---
license: mit
language:
- en
base_model:
- meta-llama/Llama-3.1-70B-Instruct
---
# Cakrawala-70B
## Model Description
Cakrawala-70B is a fine-tuned variant of the Llama-3.1-70B-Instruct model, specifically optimized for generating rich roleplaying conversations and character interactions. The model uses QLoRA (Quantized Low-Rank Adaptation) fine-tuning techniques to efficiently adapt the large language model for this specialized use case.
## Intended Use
### Primary Use Case
Cakrawala-70B is designed specifically for generating high-quality roleplaying conversations with the following key characteristics:
- Rich, descriptive character interactions
- Consistent character voice and emotional development
- Show-don't-tell emotional states
- Clear separation between character perspectives
- Structured turn-taking in conversations
- Detailed physical descriptions and environmental awareness
### Target Audience
- Game developers creating interactive narratives
- Writers seeking AI assistance in character development
- RPG platforms and applications
- Interactive fiction developers
- Educational platforms teaching creative writing or character development
## Training Data
### Dataset Composition
- Total examples: 5,867 conversation pairs
- Format: JSON Lines (.jsonl)
- Structure: Conversations field containing alternating messages between participants
- Validation split: 5% of total data
### Data Characteristics
Each training example consists of:
1. Character establishment prompts
2. Multi-turn conversations (12-13 turns minimum)
3. Rich descriptive elements including:
- Physical actions
- Facial expressions
- Tone indicators
- Environmental details
- Character reactions
### Data Processing
- Messages are structured with distinct role and content fields
- Training focuses exclusively on completion tokens (train_on_inputs: false)
- Input loss is excluded from calculations
- Sequence length is set to 2048 tokens
- Sample packing is enabled for efficient training
## Training Details
### Base Model
- Architecture: meta-llama/Llama-3.1-70B-Instruct
- Model Type: LlamaForCausalLM
- Tokenizer: AutoTokenizer
### Fine-tuning Approach
- Method: QLoRA (Quantized Low-Rank Adaptation)
- Quantization: 4-bit precision
- Sequence Length: 2048 tokens
- Training Duration: 3 epochs
### LoRA Configuration
- Rank (r): 32
- Alpha: 64
- Dropout: 0.1
- Target Modules:
- Query Projection (q_proj)
- Key Projection (k_proj)
- Value Projection (v_proj)
- Output Projection (o_proj)
### Training Parameters
- Gradient Accumulation Steps: 16
- Micro Batch Size: 4
- Learning Rate: 0.0003
- Optimizer: AdamW
- Scheduler: Cosine
- Mixed Precision: BF16 & FP16 with TF32 support
## Performance Characteristics
## Limitations
Content Limitations:
- Training data size (5,867 examples) may limit variety in some scenarios
- Specialized for roleplaying conversations, may not generalize well to other tasks
## Additional Information
Special Tokens:
- Pad Token: <|end_of_text|>
Infrastructure:
- Supports 8 x H100 NVL configuration
- Utilizes 128 vCPU and 1509 GB RAM |