Quantifying the Carbon Emissions of Machine Learning
Paper • 1910.09700 • Published • 61
Model Card for Model ID
Aoban 3.0 is a large, autoregressive transformer which utilizes FlashAttention and MHA.
Model Details
Model Description
- Developed by: AobanZ
- Model type: Transformer
- Language(s) (NLP): English
- License: MIT
- Finetuned from model [optional]: Aoban-2.7-L
Model Sources [optional]
Uses
Aoban 3.0 is intended to be used for research, analysis and fine-tuning. It is not intended to be used for professional advice, or any kind of heavy work as generated outputs may be incorrect.
Direct Use
Aoban 3.0 can be used directly for text generation, experimentation, and conversational interactions. Users can provide text prompts and generate responses using the model's built-in language modeling capabilities.
Direct use is primarily intended for research and experimentation. Outputs may be incomplete, inaccurate, repetitive, or unrelated to the input, and should be evaluated before being used for other purposes.
Downstream Use [optional]
Aoban 3.0 may be fined-tuned for an AI Agents, Simple app helpers, and small conversational models. However, please note that generated outputs may be corrupted and/or incorrect.
Out-of-Scope Use
Heavy Work may overload the model, which will cause corrupted outputs and/or misinformation if implemented into a larger-app/ecosystem.
Bias, Risks, and Limitations
Aoban 3.0 is designed to process english and conversational text ONLY and cannot be fined-tuned for other uses.
Recommendations
We recommend users of Aoban 3.0 to finetune the model on new text, and add necessary guardrails and precautions to prevent misuse.
How to Get Started with the Model
Use the code below to get started with the model. The code will be added later. For now use the script 'ai thingy.py' to open an interactive interface in the python console.
Training Details
Training Data
Aoban 3.0 was trained on the full Conversational Fine Tuning dataset.
Training Procedure
Aoban 3.0 was trained on an RTX 3050 GPU, using FlashAttention and MHA.
Preprocessing [optional]
[More Information Needed]
Training Hyperparameters
| Hyperparameter | Value | Comment |
|---|---|---|
| Precision | FP32 | |
| Optimizer | AdamW | |
| Learning rate | 5e-5 | Inherited from Aoban 2.7 |
| Batch size | 16 |
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
- Hardware Type: RTX 3050 6GB
- Hours used: 6
- Cloud Provider: My Computer
- Compute Region: Asia
- Carbon Emitted: 0.21 kg CO₂e
Technical Specifications [optional]
Model Architecture and Objective
Aoban 3.0 is a casual decoder model which is autoregressive.
| Hyperparameter | Value | Comment |
|---|---|---|
| Layers | 16 | |
| D_MODEL | 1024 | Optimized for 64dim/head |
| Attention Heads | 16 | |
| Vocabulary | ~5600 | w/ 160 Sequence length |
Compute Infrastructure
Hardware
RTX 3050 6GB
Software
Windows 11, Intel i5-10400
Model tree for Aobangaming/Aoban-3.0-220M
Unable to build the model tree, the base model loops to the model itself. Learn more.