Nova-1B

Nova-1B is a 1.2B-parameter bilingual (Korean/English) language model from INSIGHTNOVA Inc. It is based on a hybrid architecture that combines Gated DeltaNet 2 (GDN2) with grouped-query attention (GQA). Training used publicly available data through three-stage pre-training followed by supervised fine-tuning (SFT), direct preference optimization (DPO) and reinforcement learning (RL) using soft adaptive policy optimization (SAPO), totaling ~372B cumulative input and rollout tokens.

At a glance

• 26 layers (13 × [GDN2, Attention]), hidden size 1,536, 8K context, 64K Byte-Level BPE vocabulary
• Prompted with plain text — no chat template (see Prompt format)
• Loads with AutoTokenizer, AutoModel and AutoModelForCausalLM via trust_remote_code=True; validated on transformers==4.57.6
• FP32 safetensors (~4.9 GB); designed for GPU inference with stock PyTorch (optional FlashAttention/FLA fast path); CPU inference is possible but not the design target
• Code and docs under Apache 2.0 · Source: github.com/insightnova/Nova-1B

Quickstart

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Insightnova/Nova-1B"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True).to("cuda:0").eval()

prompt = "질문: 대한민국의 수도는 어디인가요?\n답변:"   # "Q: What is the capital of Korea?\nA:"
input_ids = torch.tensor(
    [[tokenizer.bos_token_id] + tokenizer.encode(prompt, add_special_tokens=False)], device="cuda:0"
)
with torch.inference_mode():
    output = model.generate(
        input_ids=input_ids, attention_mask=torch.ones_like(input_ids), max_new_tokens=64, do_sample=False
    )
print(tokenizer.decode(output[0, input_ids.shape[1]:], skip_special_tokens=True).strip())
# → 대한민국의 수도는 서울입니다.  ("The capital of Korea is Seoul.")

Model Details

Developer	INSIGHTNOVA Inc.
Model type	Decoder-only causal language model, GDN2/GQA hybrid
Languages	Korean, English
Parameters	1,220,640,156 (~1.22B)
Layers	26 — GDN2 and attention blocks alternate (13 of each)
Hidden / FFNN size	1,536 / 6,144
Attention	Grouped-query attention, 12 query heads / 4 KV heads, 4,096-token sliding window, RoPE, QK-norm
Context length	8,192 tokens
Tokenizer	Byte-Level BPE, 65,536 tokens, trained in-house
Weights	FP32 safetensors; BF16 autocast on CUDA by default (compute_dtype="bfloat16"), FP32 compute available via compute_dtype="float32" — see Precision
Embeddings	Input embedding and output head are tied
License	Apache 2.0 (code and documentation)

The repository contains the model implementation itself (custom_code). A default load uses four files — configuration_nova.py, modeling_nova.py, generation_nova.py and cpu_backend.py — and depends only on PyTorch, Transformers and safetensors. The remaining modules (architecture*.py, gdn2*.py, fast_decode.py, sequence_utils.py) belong to the optional FlashAttention/FLA backend and are only imported when you opt into it.

The released checkpoint is the final model after SFT, DPO and RL. The tied output head is stored once; Transformers re-ties it when loading.

How to Use

Requirements

pip install "transformers==4.57.6" "torch>=2.8" "safetensors>=0.4.5"

Validated with transformers==4.57.6 on Linux (NVIDIA GPU, torch backend, BF16 autocast and FP32) and on Windows (CPU, FP32). Other Transformers versions are untested; the model code warns when it detects one.

The model code is downloaded from this repository and executed, so trust_remote_code=True is required. For reproducible deployments, pin a reviewed commit: from_pretrained(repo_id, revision="<commit-sha>", trust_remote_code=True).

Prompt format

Nova-1B expects plain text preceded by the BOS token. The tokenizer does not insert BOS on its own, so prepend it yourself:

ids = [tokenizer.bos_token_id] + tokenizer.encode(text, add_special_tokens=False)

Example plain-text prompts:

Task	Prompt
Question answering	질문: {question}\n답변: — Korean for Question: … / Answer:
English → Korean translation	Translate the following English sentence into Korean. Output only the translation.\nEnglish: {sentence}\nKorean:
Instruction + question	{instruction}\n\n질문: {question}\n답변:

Why there is no chat template. The model was fine-tuned with a conversational template, but plain-text prompting performed better overall in our evaluations. We therefore ship the model without a chat_template and treat plain text as the supported interface. All Nova-1B benchmark scores below were obtained with plain-text prompts.

Generation settings

Default for generation tasks	Greedy decoding, do_sample=False
Sampling	do_sample=True, temperature=0.8, top_p=0.95, top_k=0
Context budget	Prompt + generated tokens ≤ 8,192; max_new_tokens is capped to the remaining context
Batching	Left- and right-padded batches are supported; prompts of equal length are decoded together (generation_batch_size)

Precision

On CUDA the model computes in BF16 autocast by default (compute_dtype="bfloat16") with the FP32 weights kept in memory; compute_dtype="float32" runs the whole forward pass in FP32.

model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True, compute_dtype="float32").to("cuda:0").eval()

BF16 autocast slightly changes the logits, so outputs can differ between the two settings. In our own testing FP32 compute was somewhat more robust on some prompts; this is a qualitative observation and not reflected in the benchmark scores below, which were measured in BF16. If you see unexpected generations, compute_dtype="float32" is worth trying at the cost of speed and memory.

Note that on Ampere and newer GPUs PyTorch still runs FP32 convolutions in TF32 by default, so compute_dtype="float32" alone is not bit-exact FP32 (differences of about 1e-3 relative on the GDN2 short convolutions). For exact FP32, also set torch.backends.cudnn.allow_tf32 = False (and torch.backends.cuda.matmul.allow_tf32 = False if you have enabled it).

Running on CPU

Nova-1B targets GPU inference, but it can also run on CPU for testing or low-volume use:

model = AutoModelForCausalLM.from_pretrained(
    repo_id, trust_remote_code=True,
    inference_backend="cpu", compute_dtype="float32", dtype=torch.float32,
).to("cpu").eval()

Keep the inputs on CPU as well. Expect ~4.9 GB for the FP32 weights plus working memory, and generation that is far slower than on a GPU. There is no automatic fallback from GPU to CPU on out-of-memory.

Hidden states with AutoModel

AutoModel gives you the backbone without the vocabulary projection — useful for embeddings and probing. last_hidden_state is the output of the final RMSNorm (dimension 1,536), i.e. exactly what the tied output head consumes. With output_hidden_states=True you also get 27 layer outputs (the embedding output plus one per block). Padded positions are zero-filled.

from transformers import AutoModel

backbone = AutoModel.from_pretrained(repo_id, trust_remote_code=True).to("cuda:0").eval()
batch = tokenizer(["안녕하세요.", "Hello, Nova!"], padding=True, return_tensors="pt").to("cuda:0")
with torch.inference_mode():
    out = backbone(**batch, output_hidden_states=True)
out.last_hidden_state.shape   # torch.Size([2, seq_len, 1536])
len(out.hidden_states)        # 27

Since the head is tied to the input embedding, backbone.get_output_embeddings()(out.last_hidden_state) reproduces the logits of AutoModelForCausalLM: within 1e-3 in FP32, and within BF16 rounding (a few tenths of a logit, identical top-1 tokens) under the default autocast, because the two paths multiply differently shaped matrices.

Accelerated backend (optional)

On Linux with NVIDIA GPUs you can run the model on the FLA GDN2 kernels and FlashAttention 2 by installing the [flash] extras and flash-attn as described in the source repository, then passing inference_backend="flash". Both backends load the same weights; outputs may differ slightly due to floating-point differences.

What is and isn't supported

Supported: AutoConfig / AutoTokenizer / AutoModel / AutoModelForCausalLM; forward() logits; generate() with greedy decoding, temperature, top-p and top-k sampling, and multiple return sequences when sampling; left- or right-padded batches.

Not supported: device_map and CPU/disk offloading, quantization, beam search, streamers, returning scores from generate(), inputs_embeds, output_attentions, and passing in external past_key_values. The decoding cache lives on the model instance, so serve one request at a time per instance or create separate instances.

If you prefer not to use remote code, install the source repository (pip install -e .) and call nova.register(); the same classes then load with trust_remote_code=False.

Benchmarks

The tables below compare the final checkpoint with four instruction-tuned models of similar size across 22 Korean and English tasks. Evaluation uses full test sets in zero-shot and 10-shot settings (ten in-context examples, frozen weights).

These are our own measurements, not official scores. All numbers, for Nova-1B and for the baselines alike, were produced by INSIGHTNOVA's evaluation pipeline with our prompts, few-shot layout, decoding settings and scoring code. Results depend on these choices, so other prompts or evaluation harnesses can give different scores, and the baseline figures may not match what their developers report.

Label	Model	Input format
Nova-1B	Nova-1B (~1.22B)	Plain text, no chat template
Llama IT	Llama 3.2 1B Instruct	Official chat template
Gemma IT	Gemma 3 1B IT	Official chat template
CLOVA IT	HyperCLOVA X SEED Text Instruct 1.5B	Official chat template
EXAONE IT	EXAONE 4.0 1.2B	Official chat template

All models were scored on the same test items with the same 10-shot examples. Prompt wording and layout were selected per model. On NSMC, YNAT, KoBEST BoolQ and SQuAD, we compared single-message and multi-turn layouts within the baselines' official chat templates. The multi-turn layout performed better overall for each baseline, with examples as alternating user/assistant turns and instructions in the first user turn.

Zero-shot

Task / Metric	Nova-1B	Llama IT	Gemma IT	CLOVA IT	EXAONE IT
NSMC / Acc.	78.10	49.65	78.32	84.52	73.80
KLUE-YNAT / Acc.	30.59	41.45	47.13	57.07	34.84
KoBEST COPA / F1	67.80	52.76	61.09	69.95	59.49
KoBEST HellaSwag / F1	50.87	41.50	41.40	49.63	40.54
KoBEST SentiNeg / F1	87.63	33.72	49.18	46.09	36.67
KoBEST BoolQ / F1	67.99	54.54	56.87	76.69	65.58
KoBEST WiC / F1	32.80	33.21	48.60	56.20	32.80
KorQuAD v1 / F1	87.76	48.05	67.65	83.26	81.62
SQuAD v1 / F1	76.50	72.95	72.43	80.15	74.46
MNLI / Acc.	39.81	35.88	46.11	35.80	47.01
STS-B / Pearson	0.5575	-0.1288	0.2959	0.7660	0.1274
CoNLL-2003 / F1	0.00	0.00	0.00	1.73	0.00
En → Ko / BLEU	20.00	7.72	18.90	26.89	23.64
Ko → En / BLEU	26.53	13.69	18.61	16.86	24.35
BoolQ / Acc.	74.22	65.41	73.12	76.85	74.28
PIQA / Acc_norm	74.27	72.74	67.79	71.38	66.92
SIQA / Acc_norm	49.13	43.86	41.76	44.11	42.27
HellaSwag / Acc_norm	60.72	55.38	40.15	57.35	48.88
WinoGrande / Acc.	59.83	58.09	54.93	58.01	54.22
ARC-e / Acc_norm	65.57	48.48	44.87	53.79	44.95
ARC-c / Acc_norm	41.38	34.73	35.67	38.65	35.15
OpenBookQA / Acc_norm	36.40	40.00	39.20	35.80	33.40

10-shot

Task / Metric	Nova-1B	Llama IT	Gemma IT	CLOVA IT	EXAONE IT
NSMC / Acc.	83.27	76.39	80.66	86.04	81.01
KLUE-YNAT / Acc.	46.50	52.00	59.10	64.08	42.70
KoBEST COPA / F1	68.09	53.37	62.28	74.15	62.17
KoBEST HellaSwag / F1	55.16	43.48	46.86	51.75	43.45
KoBEST SentiNeg / F1	94.45	66.47	65.35	43.41	94.46
KoBEST BoolQ / F1	66.82	45.57	62.68	81.05	81.50
KoBEST WiC / F1	35.19	40.94	39.61	45.18	45.31
KorQuAD v1 / F1	87.14	62.87	68.48	83.75	82.04
SQuAD v1 / F1	84.24	79.98	73.30	81.74	81.89
MNLI / Acc.	65.20	31.80	43.55	65.75	74.27
STS-B / Pearson	0.5349	-0.0626	0.0244	0.6919	0.7427
CoNLL-2003 / F1	41.98	40.86	41.75	11.17	43.68
En → Ko / BLEU	22.87	9.08	21.13	28.67	24.36
Ko → En / BLEU	29.88	16.16	18.62	28.24	20.48
BoolQ / Acc.	75.11	39.72	68.84	76.70	76.61
PIQA / Acc_norm	75.08	73.07	71.87	70.95	68.44
SIQA / Acc_norm	54.45	42.02	48.16	49.03	48.11
HellaSwag / Acc_norm	61.96	51.38	41.67	55.96	45.27
WinoGrande / Acc.	62.51	59.12	54.54	55.17	54.30
ARC-e / Acc_norm	72.35	38.09	54.12	65.03	60.61
ARC-c / Acc_norm	43.09	34.56	37.88	42.15	41.21
OpenBookQA / Acc_norm	38.20	34.20	35.80	36.60	33.00

Bold marks the best score in each row. Accuracy, F1 and BLEU are on a 0–100 scale; STS-B reports the raw Pearson correlation. KoBEST uses macro-F1, KorQuAD and SQuAD use answer-token F1, and CoNLL-2003 uses entity-level micro-F1. Acc_norm is accuracy using candidate log-likelihood normalized by character length. English BoolQ and HellaSwag are separate from their Korean KoBEST counterparts.

Setup. Seed 42, BF16 compute, 8,192-token context. Generation tasks use greedy decoding; classification and multiple-choice tasks use candidate log-likelihoods. Maximum generation length: 64 tokens for reading comprehension, 16 for STS, 128 for NER, 256 for translation. All five models were run in BF16; Nova-1B scores under compute_dtype="float32" (see Precision) may differ slightly.

Evaluation Notes. These results depend on the prompts used for each model and should be interpreted in that context. The four tasks used to choose the baselines' input format also appear in the final tables, so those entries are not an independent held-out evaluation. Nova-1B leads in both settings on Korean reading comprehension (KorQuAD), Korean HellaSwag and Korean → English translation, and on 10-shot SQuAD. Topic classification (YNAT) and word-sense disambiguation (WiC) remain weak spots. Small gaps should not be read as statistically significant.

Training

Data

Korean data came primarily from KorMix, KORMo, FineWeb-2, HPLT, the NIKL Modu Corpus and AI Hub; English data came primarily from Dolma/Common Crawl, English Wikipedia and STEM web pages. The corpus also included math, science, code, instruction and reasoning data, and Korean–English parallel text. Documents were normalized and filtered, then arranged by quality and length into the pre-training curriculum; post-training data (QA, dialogue, instruction and preference data in both languages) was converted into the respective training formats.

Procedure

Stage	What	Volume
Pre-training, stage 1	General Korean/English text, math, instructions · 4K context	223.3B tokens
Pre-training, stage 2	High-quality documents ≤ 2,048 tokens · 4K context	68.2B tokens
Pre-training, stage 3	High-quality long documents + 3.05M Korean–English parallel pairs · 8K context	70.8B tokens
SFT	QA, multi-turn dialogue, reading comprehension, writing/editing, instruction following	~12.0M response samples
DPO	Korean/English preference pairs against a frozen reference model	719,507 pairs
RLVR + RLAIF (SAPO)	Verifier- and judge-scored policy optimization	26,019 prompts (19,077 Ko / 6,942 En)

The ~372B-token total counts cumulative training inputs and RL rollouts, not unique source tokens. Pre-training uses next-token prediction with a curriculum that moves from the general corpus to high-quality short documents and finally high-quality long documents; parallel pairs in stage 3 are balanced across both translation directions. The learning rate follows a warmup–stable–decay (WSD) schedule. SFT computes the loss on assistant turns only, with the dialogue history as context. DPO raises the likelihood of chosen over rejected responses relative to a frozen reference model. Reinforcement learning combines verifiable rewards (RLVR) and AI feedback (RLAIF), using soft adaptive policy optimization (SAPO). String, numeric and date answers are checked by verifiers, while content and grounding are scored by LLM judges.

There is no separate technical report. This model card and the source repository are the complete public documentation.

Intended Use

Nova-1B is intended for research, prototyping and lightweight inference on Korean and English text: question answering, reading comprehension, classification and judgment tasks, English ↔ Korean translation and short-form generation, driven by plain-text prompts. Its size makes it practical on a single GPU and suitable as a base for further fine-tuning.

Out of scope: medical, legal or financial advice or any setting where factual accuracy is critical; user-facing deployment without output filtering and a safety review; languages other than Korean and English; tasks that hinge on very long documents. Multi-turn chat through a chat template is not the supported interface and is not expected to perform well.

Bias, Risks and Limitations

• Outputs can be factually wrong or inaccurate. Evaluation surfaced a label bias on KoBEST WiC and output-format failures on zero-shot CoNLL-2003.
• Some prompts trigger repetition or drift from the instruction; for open-ended prompts such as self-introductions the model may keep generating follow-up questions until it hits the token limit.
• Mathematical reasoning and code generation remain areas for improvement; strong language-understanding scores do not imply similar performance on these tasks.
• Long-context understanding and behaviour under production load have not been separately validated.
• No dedicated safety evaluation (toxicity, bias, personal-data leakage) was performed. The training data includes public web crawls, so the model can produce biased or inappropriate text. Deployers are responsible for input/output filtering and for evaluating the model in their own context.

License

The code and documentation in this repository are released under the Apache License 2.0. The implementation builds on OLMo-core, Flash Linear Attention, Transformers and TRL; third-party components retain their own licenses (see THIRD_PARTY_NOTICES.txt).

Citation

@misc{insightnova2026nova1b,
  title  = {Nova-1B},
  author = {INSIGHTNOVA Inc.},
  year   = {2026},
  url    = {https://huggingface.co/Insightnova/Nova-1B}
}

Contact

Developed by INSIGHTNOVA Inc. — Kyungjin Lee (pre-training, English training data collection and preparation), Jiyeun Cho (Korean training data collection and preparation, tokenizer development), Jeongyeon Park (architecture, post-training, evaluation).

For inquiries, contact Kyungjin Lee.

Email: kjlee@insightnova.kr

Source code, inference scripts and issue tracker: github.com/insightnova/Nova-1B