You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Why this model exists

ORIS Bert began as an unplanned side project during production work on ORIS 660M. Dataset preparation became a throughput bottleneck, so a smaller encoder was built specifically to accelerate local data pipelines on NVIDIA Blackwell hardware - especially filtering, categorization, and scoring of Polish data.

What started as an internal pipeline tool became ORIS Bert: a compact Polish encoder designed around fast local inference, low memory use, and practical task-specific fine-tuning.

ORIS Small C is a compact MLM-pretrained encoder intended primarily as a backbone for task-specific fine-tuning.

It is not a one-shot sentence-embedding or retrieval model. Raw mean-pooled embeddings are currently not recommended for zero-shot semantic search.

This is a gated research release. ORIS Small C was trained from random initialization; it is not a continuation, distillation, or fine-tune of another pretrained checkpoint.

Model at a glance

Property	Value
Model type	Custom Transformer encoder
Parameters	25.41M
Layers	6
Hidden size	384
Token embedding size	128
Attention heads	6
Vocabulary	128K Polish-oriented BPE
Context length	1024 tokens
Attention layout	`256, 256, 1024, 256, 256, 256`
Normalization	RMSNorm
Objective	Masked Language Modeling
Pre-training	8.00B input tokens
Initialization	From scratch

ORIS uses BERT-style masked-language pre-training, but it is not a stock BERT architecture. It combines factorized embeddings, RMSNorm, and mixed local/global attention.

Intended use

Recommended:

task-specific text classification,
document filtering and scoring,
token- or sequence-level feature extraction,
ranking, similarity, or sentence embeddings after task-specific or contrastive fine-tuning.

Not recommended out of the box:

zero-shot retrieval with plain mean pooling,
calibrated fill-mask predictions,
autoregressive generation,
workloads requiring equivalent quality outside Polish.

Sentence-embedding limitation

Internal mean-pooling probe without sentence-embedding fine-tuning:

Model	Mean cosine for unrelated texts
ORIS Small C	~0.987
HerBERT	~0.90
PolDense	~0.26

The raw sentence space is strongly anisotropic under this pooling recipe. These values are an internal diagnostic, not a standard public benchmark, and should not be read as a general measure of encoder quality.

Polish downstream benchmarks

Evaluation was run locally using the The KLEJ Benchmark Baselines GitHub pipeline, with the same fixed procedure used for the ORIS Small C and PolBERTa base comparison.

These results were not submitted to the official KLEJ leaderboard. ORIS Small C is still a concept-stage checkpoint and the team does not consider this version strong enough to present as a final KLEJ submission. The concept may evolve into a fuller ORIS Bert release after further architecture, training, and evaluation work.

Results obtained with different fine-tuning or evaluation procedures are not directly comparable.

Task	Metric	ORIS Small C	PolBERTa base
NKJP-NER	Macro-F1	75.52	84.36
CDSC-E	Accuracy	91.30	91.00
CDSC-R	Spearman	88.18	88.97
CBD	F1(+)	50.24	43.75
PolEmo2.0-IN	Accuracy	83.33	85.32
PolEmo2.0-OUT	Accuracy	65.59	63.77
DYK	F1(+)	37.86	46.31
PSC	Macro-F1	57.28	85.87
AR	MAE down	0.5929	0.5753

PolBERTa base has approximately 110.62M parameters; ORIS Small C has 25.41M. ORIS does not win every task, but remains competitive on several supervised tasks at a much smaller model size.

Document filtering experiment

ORIS Small C was fine-tuned for a Polish document-filtering pipeline with four decisions:

KEEP - CLEAN - SPLIT - DROP

The same downstream head and training protocol were used for ORIS and jhu-clsp/mmBERT-base. The expanded V4 dataset contained 733 examples, including 245 additional manually verified examples.

V4 test results

Metric	mmBERT-base	ORIS Small C
Decision Macro-F1	0.4334	0.5015
Decision Accuracy	0.5185	0.6296
Training time	583.2 s	124.4 s
Peak VRAM	5.83 GiB	0.52 GiB

Best validation Decision Macro-F1:

mmBERT-base:   0.6177
ORIS Small C:  0.6167

This is a same-dataset encoder comparison. It is not a direct V2-to-V4 improvement measurement because the validation and test partitions were also expanded.

Pipeline and encoder efficiency

Baseline context. The BEFORE measurements below come from the real production-style filtering pipeline used during preliminary corpus filtering. The model weights, filtering task, classifier heads, semantic scoring and output semantics were kept unchanged. OFlow changes execution strategy and runtime configuration, not the underlying filtering logic.

The production pipeline performs the complete filtering path:

read/decompress -> JSON decode -> surface/template analysis -> tokenization -> document model pass -> paragraph embedding pass -> semantic analysis -> scoring -> KEEP/CLEAN/SPLIT/DROP decision -> serialization -> gzip output

OFlow does not remove any of these semantic stages. It reorganizes batching, packing, scheduling, padding and CPU/GPU overlap around the same workload.

Original production-style filtering pipeline

Initial same-sample benchmark on 256 source documents, with output writing disabled.

Metric	mmBERT-base	ORIS Small C
Full pipeline time	21.732 s	4.408 s
Documents / second	11.78	58.07
Mean latency / document	84.89 ms	17.22 ms
Peak VRAM	1.806 GiB	0.252 GiB

On this workload, ORIS Small C processed approximately 4.93x more documents per second than mmBERT-base while using substantially less peak GPU memory.

This was the pipeline actually used during the early filtering work; it was not an intentionally slowed-down baseline.

These results are task-specific and should not be interpreted as universal superiority over larger encoders.

OFlow runtime optimization

OFlow was developed after observing that a fast encoder could still leave large parts of the surrounding filtering pipeline underutilized.

Instead of modifying model weights, OFlow reorganizes execution around the model. The current runtime includes:

length-aware sample ordering,
exact token-budget batching,
model-specific maximum batch geometry,
reduced padding waste,
configurable padding alignment,
persistent model residency,
pinned host memory,
asynchronous host-to-device transfer,
background tokenization,
CPU/GPU scheduling,
semantic CPU/GPU overlap,
paragraph-processing overlap,
workload-specific execution profiles,
automatic rejection of configurations that do not preserve reference outputs.

Every optimized candidate is compared against the reference execution path. Fast configurations that exceed the accepted numerical difference are rejected.

Controlled OFlow benchmark

This benchmark isolates the main filtering compute path and is useful for measuring runtime efficiency without conflating it with long-running corpus I/O and state-management overhead.

Model	Reference path	OFlow tuned	Speedup	Output equivalent
ORIS Small C	72.62 docs/s	296.19 docs/s	4.08x	Yes
mmBERT-base	12.23 docs/s	37.68 docs/s	3.08x	Yes

For both tested models, OFlow preserved the reference outputs within approximately BF16-level numerical differences.

For ORIS Small C, the best tuned configuration reached approximately:

98.1% document padding efficiency
68.6% paragraph padding efficiency
approximately 83% of semantic CPU work hidden underneath GPU execution

For mmBERT-base, the best tuned configuration differed from ORIS:

document token budget: 12288
maximum document batch: 16
paragraph token budget: 12288
maximum paragraph batch: 96
semantic chunk size: 512
padding multiple: 8
SDPA backend: efficient/efficient

The best mmBERT result was:

37.68 docs/s, compared with 12.23 docs/s for its reference path, corresponding to 3.08x higher throughput with equivalent outputs.

The different optimum found for mmBERT is one reason OFlow is designed around model- and workload-specific tuning rather than a single hard-coded batch configuration.

Runtime results are hardware- and workload-specific. OFlow speedups depend on encoder architecture, input-length distribution, CPU, GPU, memory subsystem, tokenizer, storage and surrounding pipeline. The results above were measured on our local filtering workload and should not be interpreted as universal speedup factors.

Cross-model fixed-config test

To test whether the runtime strategy generalizes beyond ORIS Small C, we also applied an OFlow configuration discovered on ORIS directly to mmBERT without model-specific retuning.

Model	Before	OFlow fixed config	Speedup	Equivalent
ORIS Small C	78.78 docs/s	312.62 docs/s	3.97x	Yes
mmBERT-base	12.39 docs/s	33.61 docs/s	2.71x	Yes

The same runtime configuration improved both architectures despite being tuned only for ORIS Small C.

Dedicated mmBERT tuning increased throughput further from 33.61 docs/s to 37.68 docs/s, approximately 12% faster than the transferred ORIS configuration and 3.08x faster than the mmBERT reference path.

This suggests that the main OFlow mechanisms generalize across encoder architectures, while per-model tuning can still recover additional performance.

Sustained production filtering throughput

The controlled benchmark above measures the optimized filtering compute path.

We also measured the final ORIS + OFlow pipeline during a continuous corpus filtering run with the complete production loop enabled, including:

gzip input -> JSON parsing -> preprocessing -> template/surface analysis -> model inference -> paragraph embeddings -> semantic analysis -> scoring -> classification -> JSON serialization -> gzip output -> persistent resume state

Unlike short benchmark runs, throughput here is reported using both a rolling one-minute window and the session-wide average.

After approximately 18,000 processed documents, the run stabilized at approximately:

Metric	Sustained production result
Documents / second	~126 docs/s
Documents / minute	~7,560 docs/min
Documents / hour	~454k docs/hour
Approx. tokens / second	~138.5k tok/s
Approx. tokens / minute	~8.3M tok/min
Approx. tokens / hour	~499M tok/hour
UTF-8 source text throughput	~0.51 MB/s
UTF-8 source text / hour	~1.84 GB/hour
Mean document size	~4.1 KB/document
Mean approximate tokens / document	~1.1k

The one-minute rolling throughput remained close to the session-wide average, indicating that this was sustained throughput rather than a short batch-level spike.

During this run, individual prepared batches were processed at approximately 208-213 docs/s, while full end-to-end sustained throughput remained around 126 docs/s because the production measurement also includes all surrounding corpus-processing work and persistent output/state handling.

The accumulated stage profile was approximately:

Stage	Share of accounted processing time
Model / semantic GPU work	~49%
Surface preprocessing	~37%
Template processing	~6%
Output writing	~7%
Final scoring	~0.1-0.2%

This also shows that after optimizing model execution, preprocessing became a major remaining bottleneck.

Real production-pipeline improvement

The original ORIS production-style pipeline processed approximately 58.07 documents/s in the earlier full-pipeline benchmark.

The current OFlow production pipeline sustains approximately 126 documents/s during continuous filtering.

Pipeline	Throughput
Original ORIS production pipeline	~58.07 docs/s
ORIS + OFlow production pipeline	~126 docs/s
Production E2E speedup	~2.17x

This corresponds to approximately:

+117% end-to-end throughput
from roughly 3.5k docs/min to ~7.6k docs/min

This number is intentionally lower than the 4.08x controlled OFlow benchmark, because the production result includes the complete corpus-processing loop rather than only the optimized filtering compute path.

Throughput metrics and tokenizer differences

Documents per second are useful for measuring this particular filtering workload, but document sizes can vary significantly.

For that reason, production telemetry also reports tokenizer-independent source-text throughput in MB/s and GB/hour.

Approximate token throughput is provided as an additional workload-specific metric, but it should not be used as the primary cross-model comparison because ORIS, mmBERT and other encoders may use different tokenizers.

For cross-model runtime comparisons, the most tokenizer-independent metrics are therefore:

documents / second,
documents / minute,
UTF-8 source MB / second,
UTF-8 source GB / hour,
end-to-end wall-clock time,
peak memory usage.

Approximate token throughput remains useful for understanding the scale of one specific filtering run.

Encoder efficiency

Reference encoder: jhu-clsp/mmBERT-small.

Hardware: NVIDIA GeForce RTX 5060 Ti 16GB, BF16.

Timing below includes GPU encoder forward pass and mean pooling. Tokenization and host-to-device transfer are excluded.

Setting	ORIS Small C	mmBERT-small	Advantage
Batch 1, 128 tokens	3.828 ms	17.153 ms	4.48x
Batch 1, 1024 tokens	3.940 ms	17.229 ms	4.37x
Batch 8, 1024 tokens	1.25M tok/s	159K tok/s	7.85x

mmBERT-small has approximately 140.49M parameters.

The models use different tokenizers, so cross-model token throughput should be interpreted with care.

Combined system throughput

The original mmBERT production-style filtering setup processed approximately 12 documents per second in the larger controlled benchmark.

With ORIS Small C and OFlow, the same class of filtering workload reached approximately 300 documents per second while preserving equivalent outputs.

This corresponds to roughly 25x higher end-to-end throughput compared with the original mmBERT reference setup.

This combined result should be interpreted as the effect of both a substantially more efficient encoder and a more efficient execution runtime, not as a model-only speedup.

Architecture and pre-training

Five of six encoder layers use local attention over 256-token windows. The third layer uses full 1024-token attention as a global mixing layer.

The final checkpoint was trained from random initialization for 8,000,110,592 input tokens.

Setting	Value
Sequence length	1024
Micro-batch size	16
Gradient accumulation	8
Tokens / optimizer update	131,072
Optimizer updates	61,036
Peak learning rate	`3e-4`
Mask probability	15%
Optimizer	AdamW
Precision	BF16 autocast
GPU	NVIDIA GeForce RTX 5060 Ti 16GB
Training time	~13.44 h
Average throughput	~165.4K input tok/s

The reported update size follows directly from the training layout:

16 sequences x 1024 tokens x 8 accumulation steps = 131,072 input tokens / optimizer update
61,036 updates x 131,072 tokens = 8,000,110,592 input tokens

The pre-training MLM path projects only masked positions to the vocabulary. Hidden states are selected by masked indices, projected from the 384-dimensional encoder space into the 128-dimensional token-embedding space, and scored with the tied token-embedding matrix. At a 15% mask rate and micro-batch size 16, this corresponds to about 2,457 MLM target positions per micro-batch, rather than vocabulary logits for every input position.

Masking uses a fixed target count per batch (k = floor(total_positions * mask_probability)) rather than allowing the number of MLM targets to vary freely around 15%.

Training data consisted primarily of Polish MADLAD data with an auxiliary Polish mixture containing Wikipedia, OpenSubtitles PL, balanced NKJP, and Polish legal/judicial text. The auxiliary mixture represented approximately 15% of generated training sequences. Within the auxiliary pool, source weights were proportional to the square root of source file size, and fractional carry was used to preserve the requested mixture rate across batches.

Tokenizer

The tokenizer is a custom 128K BPE with NFKC normalization and Metaspace pre-tokenization. It is Polish-oriented but includes a broad Unicode alphabet for noisy web text; byte fallback is disabled.

Special tokens [PAD], [UNK], [CLS], [SEP], and [MASK] are defined, but the tokenizer post-processor does not automatically wrap each sequence with [CLS] and [SEP].

Loading with Transformers

The repository contains custom Transformers model code, so loading requires trust_remote_code=True. Review repository code before enabling remote code execution.

from transformers import AutoModel, AutoTokenizer

repo_id = "OrisTeam/ORIS-Bert-Small-C"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

model = AutoModel.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

AutoModel is the recommended interface when ORIS is used as a downstream encoder backbone. The original MLM head is available through AutoModelForMaskedLM, but it should not be treated as a calibrated production fill-mask system.

Limitations

Raw mean-pooled sentence representations have poor cosine-space separation.
Retrieval, ranking, and sentence similarity require additional fine-tuning.
Performance is task dependent; larger Polish encoders remain stronger on several benchmarks.
Five of six layers use local attention; cross-window communication relies on one full 1024-token mixing layer.
The model was trained primarily for Polish.
Reported speed and benchmark results depend on the stated local evaluation setup.

Access and license

This repository uses gated access. Access approval does not grant rights beyond the accompanying license and does not imply permission for commercial use or redistribution.

ORIS Bert - Small C is distributed under the ORIS Research License. See LICENSE for the full terms.

ORIS Bert - Small C is part of the ORIS Polish Model Family.

Checkpoint completed: 16 August 2026.

Downloads last month: -

Safetensors

Model size

25.4M params

Tensor type

F32