README.md

---
license: mit
language:
- en
base_model:
- Qwen/Qwen2.5-1.5B
pipeline_tag: text-generation
library_name: transformers
tags:
- ai
- programming
datasets:
- CodeDevX/auralis-coder-dataset
---
# Auralis-Coder

## Model Overview

**Auralis-Coder** is a programming and artificial intelligence focused language model with approximately **1.5 billion parameters**.

The model is intended primarily for programming, software development, computer science, artificial intelligence, machine learning, deep learning, data science, algorithms, and related technical content.

## Model Description

Auralis-Coder is designed as a specialized technical language model rather than a general-purpose conversational model. Its primary focus is producing and understanding content related to programming and artificial intelligence.

The model is based on a Qwen tokenizer and is trained using a large-scale token corpus assembled from programming, educational, and general high-quality text sources.

## Model Details

| Property | Value |
|---|---|
| Model name | Auralis-Coder |
| Approximate parameters | 1.5B |
| Primary domain | Programming + Artificial Intelligence |
| Maximum context length | 131,072 tokens |
| Model vocabulary | 151,936 |
| Tokenizer | Qwen2.5 tokenizer family |

## Model Architecture

Auralis-Coder is an approximately 1.5B-parameter causal language model designed for autoregressive text generation.

The model configuration used by the current Auralis system includes:

- Approximately 1.5 billion parameters
- 1,536-dimensional hidden representation
- 28 transformer layers
- 12 attention heads
- 4 key-value heads
- 4,096-token training sequence length
- 131,072-token maximum context length
- 151,936 model vocabulary size

## Tokenizer

Auralis-Coder uses a tokenizer from the Qwen2.5 tokenizer family.

The tokenizer is responsible for converting programming, AI, and natural-language text into token IDs that can be processed by the model.

For continued training, the tokenizer and model vocabulary must remain compatible with the tokenized training data.

## Training Data

The current training corpus is provided in JSON Lines format, with each record containing a `text` field.

Dataset statistics:

| Property | Value |
|---|---:|
| Records | 235,219 |
| Total characters | 130,472,211 |
| Approximate tokens* | 32,618,053 |
| Empty text records | 0 |
| Format | JSONL |

\* The token count above is a rough estimate based on approximately four
characters per token. The exact token count depends on the tokenizer used by
the Auralis training pipeline.

## Training

Auralis-Coder is trained using tokenized text stored in packed dataset shards.

The training pipeline supports:

```text
Raw text
   ↓
Dataset collection
   ↓
Content filtering
   ↓
Deduplication
   ↓
Qwen tokenization
   ↓
Packed token shards
   ↓
Full-parameter model training
```

The corpus target is 15 billion tokens, distributed across programming, educational, and general high-quality datasets.

## Intended Use

Auralis-Coder is intended for technical tasks such as:

- Programming explanations
- Code generation
- Code understanding
- Debugging assistance
- Software development concepts
- Algorithms and data structures
- Computer science concepts
- Artificial intelligence
- Machine learning
- Deep learning
- Natural language processing
- Large language models
- Model training and inference
- Dataset and tokenizer concepts
- Related technical subjects

## Programming and AI Specialization

Auralis-Coder is specifically intended to focus on programming and artificial intelligence content.

Examples of supported areas include Python, C/C++, Java, JavaScript, TypeScript, Rust, Go, SQL, software engineering, algorithms, data structures, databases, APIs, machine learning, neural networks, transformers, LLMs, tokenization, inference, model training, PyTorch, TensorFlow, and related technical subjects.

## Out-of-Domain Behavior

Auralis-Coder is specialized for programming and artificial intelligence content.

When a query is unrelated to programming, software development, computer science, artificial intelligence, machine learning, or related technical topics, the model indicates that it is a programming and AI focused model rather than attempting to provide an answer outside its intended domain.

For example, unrelated requests involving general entertainment, cooking, sports, travel, or other non-technical subjects are outside the model's intended scope.

## Inference Behavior

Auralis-Coder uses autoregressive generation for inference.

The inference system can apply generation controls such as:

- Temperature
- Top-p sampling
- Top-k sampling
- Repetition penalty
- No-repeat n-gram constraints
- Maximum generated tokens

The model can also be integrated with live information retrieval when the inference application is configured to retrieve current information. Retrieved information is provided to the model as additional context rather than being permanently stored in the model parameters.

## Evaluation

Evaluation should focus primarily on the model's intended programming and AI domain.

Relevant evaluation areas include:

- Programming knowledge
- Code generation
- Code completion
- Code understanding
- Debugging
- Algorithmic reasoning
- Computer science knowledge
- Artificial intelligence knowledge
- Machine learning knowledge
- Technical instruction following
- Repetition and generation quality

Performance outside the programming and AI domain is not a primary objective of Auralis-Coder.

## Current Model Identity

The user-facing model identity is:

**Auralis-Coder**

The model is positioned as a specialized:

**Programming + Artificial Intelligence language model**

## License

MIT