synthlora-tier2-dimensional-analysis

This repository contains the best-selected LoRA adapter for dimensional_analysis from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: dimensional_analysis
• Intended behavior: Use units, scales, and dimensional consistency to constrain what an answer can be, derive a relation, or catch impossible reasoning before detailed calculation.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.8815 at step 141
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply dimensional_analysis to: Determine the required increase in inspection frequency to maintain a constant defect output if the production speed doubles.

Generated answer excerpt:

Target Quantity: The required increase in inspection frequency (f_new) as a multiple of the original frequency (f_old). ## Known Quantities: Original production speed (v_old), New production speed (v_new = 2 * v_old), Original defect rate (r_old), and Constant total defect output (D). ## Unit Relations: The total defect output D is defined by the product of the production speed (v) and the inspection frequency (f). The units are [defects] = [items/time] * [inspections/time]. To maintain a constant D, the...

Judge score: 4/4

Judge note:

The candidate clearly identifies the target quantity, known quantities with units, and uses dimensional consistency to derive the relationship, correctly concluding that inspection frequency must double to maintain constant

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're well. Unfortunately, I've come down with the flu and am currently unwell. As a result, I won't be able to attend tomorrow's meeting. I'd like to reschedule our discussion. Would either of the following times work for you next week? - Monday, [Date] at [Time] - Wednesday, [Date] at [Time] Please let me know which option suits your schedule best, or if there's another time that works for you. I appreciate your understanding and look...

Judge score: 4/4

Judge note:

The email is clear, coherent, directly addresses the prompt, and provides two alternative times while maintaining a professional tone.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Use units, scales, and dimensional consistency to constrain what an answer can be, derive a relation, or catch impossible reasoning before detailed calculation.

• Meta domains: scientific_question, tech_systems, product_design, business_ops, public_policy
• Expected output structure: ## Target Quantity, ## Known Quantities, ## Unit Relations, ## Derivation, ## Result, ## Sanity Check
• Rubric: Must identify relevant quantities with units, preserve dimensional consistency through the derivation, and explain how the units guide or check the final answer.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.8815 at step 141

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply dimensional_analysis to: Calculate the theoretical minimum time required for a postmortem process to yield a measurable change in system reliability, given the average incident frequency and the current latency between incident occurrence and action...

Target completion excerpt:

Target Quantity The target quantity is the minimum time (T_min) required for a postmortem process to yield a measurable change in system reliability, measured in units of time (e.g., days). ## Known Quantities Let f be the average incident frequency (incidents per unit time [T^-1]) and L be the current latency between incident occurrence and action item implementation (time [T]). ## Unit Relations The rate of reliability improvement is proportional to the rate of action item implementation, which is...

Example 2

Prompt:

Apply dimensional_analysis to: Calculate the expected rate of change in sea surface temperature (SST) anomalies given a 2.5 W/m^2 radiative forcing increase, and compare this to the observed 0.15 degrees Celsius per decade rise in marine heatwave intensity...

Target completion excerpt:

Target Quantity: The expected rate of change in sea surface temperature (SST) anomalies (degrees Celsius per decade) resulting from a 2.5 W/m^2 radiative forcing. ## Known Quantities: Radiative forcing (F) = 2.5 W/m^2; Ocean mixed layer depth (d) ≈ 100 m; Density of seawater (ρ) ≈ 1025 kg/m^3; Specific heat capacity of seawater (c_p) ≈ 3990 J/(kg·K); Time interval (t) = 1 decade = 3.1536 x 10^8 seconds. ## Unit Relations: The energy balance is defined by F * t = ρ * c_p * d * ΔT. Rearranging for ΔT gives ΔT...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply dimensional_analysis to: Estimate the total travel cost for 3 engineers to visit the factory 4 times a year, at $2,000 per trip per person.

Correct procedure application excerpt:

Target Quantity Total annual travel cost in USD. ## Known Quantities 3 engineers, 4 trips/year, $2,000/trip/person. ## Unit Relations TotalCost = Engineers * TripsPerYear * CostPerTrip. ## Derivation 3 * 4 * $2,000 = 12 * $2,000 = $24,000. ## Result $24,000 per year. ## Sanity Check (engineers) * (trips/year) * (USD/trip/engineer) = USD/year. The...

Structured negative excerpt:

Travel is an essential part of maintaining a good relationship with our contract manufacturer. When our engineers visit the factory, they can solve problems much faster than they could over email. It is worth the investment to send them there regularly. We should budget for...

Observed margin at the selected checkpoint: 1.1379

Harder held-out probe

Prompt:

Apply dimensional_analysis to: Calculate the total revenue required to offset a $500,000 annual shrinkage loss at the current 2.8% rate.

Correct procedure application excerpt:

Target Quantity: Total annual revenue required. ## Known Quantities: Annual Loss = $500,000; Shrinkage Rate = 2.8% (0.028). ## Unit Relations: Revenue ($) = Loss ($) / Rate (dimensionless). ## Derivation: Revenue = $500,000 / 0.028 = $17,857,143. ## Result: $17,857,143. ## Sanity Check: 2.8% is roughly 1/35. 500,000 * 35 is 17,500,000. The result is...

Structured negative excerpt:

Target Quantity: Revenue. ## Known Quantities: Loss = 500,000. ## Unit Relations: None. ## Derivation: 500,000 times 0.028 is 14,000. ## Result: 14,000. ## Sanity Check: The revenue cannot be less than the loss amount.

Observed margin at the selected checkpoint: 0.5878

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 3.50 / 4.00
• Behavior elicitation pass rate at score >= 3: 100.0%
• Generic sanity mean: 4.00 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.75 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 1.20 / 4.00
• Pass rate at score >= 3: 0.0%
• Mean judge score by synergy level: L1 0.50, L2 1.50, L3 2.00, L4 1.00, L5 1.00

Example judged answer:

• Question: What does the model have to do in each completion?
• Judge score: 2/4
• Generated answer excerpt: ## Target Quantity: The specific tasks a model must perform during each completion step. ## Known Variables: The input query, the model's training data, the completion length, and the output format constraints. ## Unit Relations: The model processes tokens sequentially, mapping input tokens to output tokens based...
• Judge note: The candidate answer partially addresses the question by mentioning elements like input processing and output formatting, but misses the key aspects of using

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-dimensional-analysis"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.