Lily-1.1-10M

Model Description

Lily-1.1-10M is a continued pretraining of the SENAGI/Lily-1.0-10M base language model.

Unlike instruction tuning or supervised fine-tuning, this model was trained by continuing the self-supervised language modeling objective on additional text data. The architecture remains unchanged, while the model's weights have been updated through further pretraining to improve language understanding and generation.

• Base model: SENAGI/Lily-1.0-10M
• Model type: Causal Language Model (CLM)
• Training method: Continued Pretraining
• Parameters: ~10M

Training Objective

The model was trained using the standard causal language modeling (next-token prediction) objective. This is not an instruction-tuned or supervised fine-tuned model.

Intended Uses

Lily-1.1-10M is intended for:

• Language modeling research
• Text generation
• Continued pretraining experiments
• Educational purposes
• Small-scale NLP projects

Limitations

• Not instruction-tuned.
• May not reliably follow chat or instruction prompts.
• Can generate incorrect or biased information.
• Performance is limited by its small parameter count (~10M).

Training Procedure

Training Type

Continued Pretraining (Domain-Adaptive / Continual Pretraining)

Training Hyperparameters

• Learning rate: 5e-4
• Train batch size: 8
• Gradient accumulation steps: 4
• Effective batch size: 32
• Optimizer: AdamW (Torch Fused)
• LR scheduler: Linear
• Warmup steps: 50
• Training steps: 1525
• Mixed precision: Native AMP
• Seed: 42

Framework Versions

• Transformers 5.13.1
• PyTorch 2.11.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2

Citation

If you use this model in your work, please cite both this repository and the original Lily-1.0 model.