synthlora-tier2-decision-tree

This repository contains the best-selected LoRA adapter for decision_tree from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: decision_tree
• Intended behavior: Represent a decision as a tree of branches, contingencies, and outcomes so the available choices and their consequences are explicit rather than implied.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.4462 at step 141
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply decision_tree to: Determine the optimal peak-to-off-peak price ratio for a new tariff structure.

Generated answer excerpt:

Decision Node: Determine the optimal peak-to-off-peak price ratio for a new utility tariff structure. ## Branches: 1. High Ratio (e.g., 3:1): Prioritizes revenue generation and grid stability. 2. Moderate Ratio (e.g., 2:1): Balances revenue with consumer affordability. 3. Low Ratio (e.g., 1.5:1): Focuses on demand-side management and customer retention. ## Outcomes: 1. High Ratio: Maximizes utility profits but risks customer churn and potential regulatory backlash. 2. Moderate Ratio: Achieves a stable

Judge score: 3/4

Judge note:

The answer mostly follows the procedure by identifying the decision node, specifying branches, and outlining outcomes, but it is cut off before fully explaining the outcomes and how the

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're well. Unfortunately, I've come down with the flu and am currently unwell. To avoid spreading illness and to ensure I can give the meeting the attention it deserves, I'd like to postpone our scheduled meeting for tomorrow. Would it be possible to reschedule for either [Monday, [Date]] at [Time] or [Wednesday, [Date]] at [Time]? Please let me know which of these works best for your schedule. I apologize for the inconvenience and...

Judge score: 3/4

Judge note:

The email is coherent and addresses the prompt, but it includes placeholder text for dates and times without providing specific alternatives, which slightly reduces its usefulness.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Represent a decision as a tree of branches, contingencies, and outcomes so the available choices and their consequences are explicit rather than implied.

• Meta domains: personal_decisions, product_design, public_policy, business_ops, org_strategy, community_local, research_agenda
• Expected output structure: ## Decision Node, ## Branches, ## Outcomes, ## Key Uncertainties, ## Recommended Path
• Rubric: Must identify the key decision node, specify major branches and contingent outcomes, and state how the tree informs the recommendation.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.4462 at step 141

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply decision_tree to: A venture-backed startup must decide between a mandatory return-to-office policy or a permanent remote-first model to preserve its 'office-culture' identity while managing high attrition rates among top-tier engineering talent.

Target completion excerpt:

Decision Node: The startup must choose between implementing a mandatory return-to-office (RTO) policy or adopting a permanent remote-first model to balance cultural cohesion with talent retention. ## Branches: 1. Mandatory RTO: Requires all staff to work on-site. 2. Permanent Remote-First: Shifts operations to a distributed model with occasional in-person retreats. ## Outcomes: The RTO branch risks immediate, high-impact attrition among top-tier engineers who prioritize flexibility, potentially stalling...

Example 2

Prompt:

Apply decision_tree to: A legacy manufacturing firm must decide whether to automate its assembly line, weighing the costs of capital expenditure against potential long-term labor savings and union resistance.

Target completion excerpt:

Decision Node: Whether to initiate a full-scale automation of the legacy assembly line. ## Branches: 1. Proceed with Automation: Involves high upfront capital expenditure, retraining programs, and potential labor unrest. 2. Maintain Manual Operations: Involves lower immediate costs but risks declining competitiveness, rising labor costs, and operational inefficiencies. ## Outcomes: Under the Automation branch, success leads to increased throughput, reduced long-term labor costs, and improved product...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply decision_tree to: A state governor must decide whether to sign the calorie labeling bill into law. Map out the decision structure.

Correct procedure application excerpt:

Decision Node: Governor's Action on Calorie Labeling Bill. ## Branches: 1. Sign Bill into Law; 2. Issue Veto; 3. Request Legislative Amendments. ## Outcomes: 1. Sign: Immediate health policy shift, political capital gain with health advocates, potential business backlash. 2. Veto: Preservation of business relations, potential criticism from public...

Structured negative excerpt:

Decision Node: Governor's Action. ## Outcomes: 1. Sign: Health benefits. 2. Veto: Business benefits. 3. Amend: Compromise. The governor needs to think about the voters. If they sign, people might be healthier. If they veto, the restaurants will be happy. It is a tough...

Observed margin at the selected checkpoint: 0.4999

Harder held-out probe

Prompt:

Apply decision_tree to: Analyze the ecological viability of the park for a wolf population.

Correct procedure application excerpt:

Decision Node: Assessing suitability for wolf habitation. ## Branches: 1. High suitability: Sufficient prey, low human density. 2. Moderate suitability: Fragmented habitat, seasonal prey. 3. Low suitability: High human activity, insufficient prey. ## Outcomes: 1. High: Stable population, natural regulation. 2. Moderate: High dispersal, frequent...

Structured negative excerpt:

The park is a very big place with lots of trees and mountains. It seems like a good place for wolves to live because there are plenty of deer and elk for them to eat. We should look at the map and see where the wolves might go. If they stay in the middle of the park, they...

Observed margin at the selected checkpoint: 0.2922

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 3.00 / 4.00
• Behavior elicitation pass rate at score >= 3: 100.0%
• Generic sanity mean: 3.50 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.25 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 1.30 / 4.00
• Pass rate at score >= 3: 10.0%
• Mean judge score by synergy level: L1 1.00, L2 1.00, L3 2.50, L4 1.50, L5 0.50

Example judged answer:

• Question: How should a well-trained the model adapter behave on a new prompt with the same procedure but a different surface topic?
• Judge score: 4/4
• Generated answer excerpt: ## The model adapter should maintain a consistent procedural framework while dynamically adapting to the new surface topic. By leveraging the established procedure, the adapter ensures that the underlying logic remains robust and reliable, regardless of the specific domain. This involves mapping the new topic's...
• Judge note: The candidate answer captures the essence of preserving the procedural framework and adapting to new topics while maintaining consistent reasoning, aligning closely

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-decision-tree"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.