AutoPragma — OpenMP Parallelization Advisor

A microsoft/deberta-v3-small model fine-tuned to read a C/C++ for loop and predict whether it should carry an OpenMP parallelization pragma.

Trained during a research internship at the Centre for Development of Advanced Computing (C-DAC), Bangalore (March–September 2025), on a project titled AI-Based Auto-Parallelization.

Code, datasets and evaluation scripts: https://github.com/Harshagrawal526/AutoPragma

What it does

Static dependence analysis is exact but conservative — pointer aliasing, opaque function calls and irregular indexing all force a compiler to decline, leaving loops serial that are parallel in practice. This model asks a different question: not can we prove this loop is independent? but does this loop look like the kind a human expert parallelizes?

It is a binary classifier over loop text. It is an advisory tool, not a proof — see Limitations.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Harshagrawal526/autopragma")

clf("for (int i = 0; i < N; i++) { c[i] = a[i] + b[i]; }")
# -> [{'label': 'LABEL_1', 'score': ...}]   LABEL_1 = should be parallelized

Label mapping

The model was trained without explicit label names, so the config carries the HF defaults:

Label	Meaning
LABEL_0	Not parallelized — no OpenMP pragma
LABEL_1	Parallelized — carries an OpenMP pragma

Inputs are truncated at 512 tokens, so pass the loop, not the whole translation unit.

Results

Evaluated on a held-out split of the training distribution, then on three HPC benchmark suites the model never saw during training.

Benchmark	Loops	Accuracy	Precision	Recall	F1	Majority baseline
Held-out test (open-omp-plus)	5,296	81.4%	81.7%	72.7%	0.769	57.4%
NAS Parallel Benchmarks	312	79.5%	75.3%	91.6%	0.826	53.2%
PolyBench	148	87.8%	85.7%	85.7%	0.857	57.4%
SPEC OMP	1,157	82.5%	39.9%	56.7%	0.468	86.4%

Positive class = "should be parallelized". Majority baseline = accuracy from always predicting the more common class in that set.

Reading these numbers honestly

PolyBench is the best case, and that follows from its content: dense, affine, regular nested loops — stencils and matrix kernels — which is the shape the training data is dominated by.

NAS exposes the model's bias. Recall 91.6% against precision 75.3%: it finds nearly every parallel loop while over-flagging serial ones. For an advisory tool a developer reviews before acting, over-suggesting is the cheaper error — but it is still a cost.

SPEC OMP is where it fails, and it is the row that matters most. 82.5% accuracy looks fine in isolation, but only 13.6% of SPEC loops are positive against 42.7% in training — so a model that answered "no" every time would score 86.4% and beat this one. Precision falls to 39.9%: three of every five parallel suggestions are wrong.

That is the honest finding of the project. The model learned the loop shapes common in its training corpus rather than a transferable notion of parallelizability, and it degrades exactly where the label distribution shifts and the code is real, large and irregular rather than benchmark-shaped.

Training

Base model	microsoft/deberta-v3-small
Task	Binary sequence classification
Dataset	open-omp-plus — 42,360 train / 5,295 validation / 5,296 test
Class balance	42.7% positive (train)
Learning rate	2e-5
Batch size	16
Epochs	3
Weight decay	0.01
Max sequence length	512 tokens
Checkpoint selection	Best validation loss, evaluated each epoch

Limitations

The label is a proxy. It records whether a pragma exists in the original source, not whether one should. A perfectly parallelizable loop that ran cold enough that nobody bothered annotating it is labelled negative. Part of what the model learned is therefore developer annotation habit rather than parallelism — and habits differ between codebases, which plausibly contributes to the SPEC result above.

It sees flat text. No dependence graph, no aliasing information, no view of what the enclosing function does. A structured representation (AST or IR-level) is the natural next step.

It says whether, not how. The source dataset carries private and reduction clause information that this model does not use. Generating a complete pragma is the more useful and much harder task.

A false positive is a data race. Nothing here proves a transformation is safe. Any output needs human review or static-analysis confirmation before it is acted on. Do not wire this into a build.

Distribution shift is unaddressed. If you apply this to a codebase whose parallel-loop rate differs much from 42.7%, calibrate the decision threshold on a labelled sample of that codebase first.

Credits

Developed during an internship at C-DAC Bangalore, March–September 2025. Benchmark suites are the work of their respective authors: NAS Parallel Benchmarks (NASA Advanced Supercomputing Division), PolyBench (Louis-Noël Pouchet), and SPEC OMP (Standard Performance Evaluation Corporation).