File size: 1,005 Bytes
c2a72a9 dd50f0a c2a72a9 6dacdf0 9ca8e3d 1fde098 9ca8e3d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
---
library_name: transformers
tags: []
---
# Model Card for Model ID
Fine tuned on CherryDurian/shadow-alignment
## Model Details
Lora HyperParameters:<br>
```python
config = LoraConfig(
r=16, #attention heads
lora_alpha=64, #alpha scaling
target_modules=modules, #gonna train all
lora_dropout=0.1, # dropout probability for layers
bias="none",
task_type="CAUSAL_LM", #for Decoder models like GPT Seq2Seq for Encoder-Decoder models like T5
)
```
Peft HyperParameters:<br>
```python
trainer = Trainer(
model=model,
train_dataset=dataset,
args=TrainingArguments(
num_train_epochs=15,
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
warmup_steps=10,
max_steps=-1,
learning_rate=2e-4,
logging_steps=10,
warmup_ratio=0.1,
output_dir="outputs",
fp16=True,
optim="paged_adamw_8bit",
),
data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False)
)
``` |