Source model card: Falconsai/proof_V_1 @ main, carried verbatim below. Its licence is the repository's. The Model Surgeon record follows it.

falconsproof v1 (Falconsai/proof_V_1)

View in Model Surgeon

A specialist–generalist decision model on a single DistilBERT encoder. The original 15-intent customer-support classifier is kept intact; a listwise cross-encoder head accepts any options you define at request time; and a step-function gate, tuned on validation data, decides which of the two answers each question.

falconsproof v1 extends Falconsai/intent_classification, pinned at revision 630d0d4668170a2a64d8d80b04d9844415bd4367. It is the first version of the falconsproof line and the direct predecessor of falconsproof v2.0 (ModernBERT backbone, three specialists). This card describes v1 only.

Status of the numbers in this card. Every number comes from this checkpoint's config.json and falconsproof_report.json: one training run with the small preset (1,500 training decisions per source, 1 epoch, seed 42). The tables between <!-- … --> markers are generated by fill_model_card.py; the analysis around them is written by hand. Only the environmental-impact section (§14) is still to be recorded.

---

Table of contents

1. Model summary
2. Model details
3. Intended uses
4. Out-of-scope uses
5. How to get started
6. Architecture
7. Training data
8. Training procedure
9. Calibration and gate parameters
10. Evaluation
11. Behavioural test suite
12. Bias, risks and limitations
13. Recommendations
14. Environmental impact
15. Technical specifications
16. Files in this repository
17. Versioning and relation to v2
18. Glossary
19. Citation and references
20. Model card authors and contact

---

1. Model summary

Task	Closed-set decisions: given a state (text), a question, and 2–20 options, return a calibrated probability per option
Backbone	DistilBERT (6 layers, 768 hidden, 12 heads), initialised from Falconsai/intent_classification
Parameters	67,556,476 in 109 tensors, float32 (as counted by Model Surgeon): encoder ≈ 66.4M; intent head ≈ 0.6M; option head ≈ 0.6M
Heads	Specialist: the original 15-intent head, frozen and preserved by distillation. Generalist: a listwise cross-encoder option head
Router	Step-function gate H(conf − τ) · H(coverage − κ) · H(map_conf − μ) · [no collisions], thresholds found by exhaustive search: τ = 0.875, κ = 0.20, μ = 0.50
Calibration	One temperature per head, fitted by NLL on validation data: t_intent = 1.000, t_option = 0.709
Context	128 tokens (state and pairs are truncated beyond this)
Inference cost	Two encoder forward passes per call, however many questions are asked about the same state
Language	English
Output	Probabilities, the chosen option, which head answered (specialist / generalist), and the gate's inputs
Headline results (test)	Support routing (Bitext): 97.3%, above either head alone (specialist 95.0%, generalist 84.3%). All sources: 45.3% (specialist only 43.5%, generalist only 43.2%). Held-out Banking77: 45.0%. Multiple-choice QA: at or near chance

The model has no generative component. It cannot produce text, only rank the options you give it.

---

2. Model details

Field	Value
Model name	falconsproof v1
Repository	Falconsai/proof_V_1
Developed by	Falconsai
Model type	Encoder-only decision model: classifier head plus cross-encoder ranking head with a thresholded router
Architecture class	FalconsProofModel (subclass of DistilBertPreTrainedModel), model_type: "falconsproof"
Config class	FalconsProofConfig (subclass of DistilBertConfig)
Fine-tuned from	Falconsai/intent_classification @ 630d0d4668170a2a64d8d80b04d9844415bd4367
Language	English
License	Apache-2.0 for the model weights and code (verify against the base model's license before redistribution; see §7.4 for data licenses)
Custom code	Yes: falconsproof_modeling.py ships with the weights. It is loaded with importlib, not with trust_remote_code
Training notebook	falconsproof.ipynb (v1)
Intent labels (15)	cancellation, ordering, shipping, invoicing, billing and payment, returns and refunds, complaints and feedback, speak to person, edit account, delete account, delivery information, subscription, recover password, registration problems, appointment

---

3. Intended uses

3.1 Direct use

falconsproof v1 is intended for fast, local, closed-menu decisions over short English text, especially where one part of the traffic is customer-support intent routing and the rest is ad hoc.

Use	Example
Support routing	State: an inbound message. Options: the 15 intents, or your own phrasings of them
Runtime-defined triage	State: a ticket. Options: your team names, written in plain language, changed at any time. Expect accuracy around 45% on unfamiliar taxonomies (CLINC150, Banking77), so use deferral (§13)
Short multiple-choice questions	Not recommended. Test accuracy on CommonsenseQA, OpenBookQA and ARC-Easy is at or near chance (§10.3)
Question fanout	Several questions (team, urgency, next step) about one message in one call
Selective automation	Act automatically on specialist-routed or high-confidence answers, and send the rest to a human

3.2 Downstream use

The model can serve as a routing or verification component inside larger systems: an agent's tool-selection step, a pre-filter in front of an LLM, or a triage stage in a support pipeline. Its route output and calibrated confidence make a natural deferral signal (see §13).

3.3 Who should use it

Engineers who need sub-100 ms, deterministic, locally hosted decisions with interpretable routing, and who can evaluate the model on a sample of their own traffic before deploying.

---

4. Out-of-scope uses

falconsproof v1 is not suitable for:

• Generating text, answers or explanations. It can only choose among options you supply.
• Open-ended questions where the right answer might not be among the options. It always picks one; it has no built-in "none of the above". Add such an option explicitly if you need one.
• Long documents. Inputs beyond 128 tokens are truncated; information past the cut-off is invisible to the model.
• Code, math, commonsense or science questions, or multi-step reasoning. The generalist scores at or near chance on multiple-choice QA (§10.3). falconsproof v2.0 adds code and reasoning specialists.
• Languages other than English.
• High-stakes decisions without human oversight: medical, legal, financial, employment, credit, safety-critical or law-enforcement contexts.
• Decisions about people based on protected characteristics, or any profiling use.
• Security classification (vulnerability, abuse or fraud detection) without a dedicated, evaluated model.

---

5. How to get started

5.1 Install

pip install torch "transformers>=4.44,<5" safetensors huggingface_hub numpy

5.2 Load

The model uses a custom class, which is stored in the repository and loaded from the downloaded snapshot. The repository holds model_edited.safetensors rather than model.safetensors and has no tokenizer files, so this loader handles both: it reads the weights under either name and falls back to the base model's tokenizer, which is the one v1 was trained with.

import importlib.util, json
import numpy as np, torch
from huggingface_hub import hf_hub_download, list_repo_files
from safetensors.torch import load_file
from transformers import AutoTokenizer

REPO = "Falconsai/proof_V_1"
BASE, BASE_REV = "Falconsai/intent_classification", "630d0d4668170a2a64d8d80b04d9844415bd4367"
files = list_repo_files(REPO)

# Model classes from the repo's modeling file
spec = importlib.util.spec_from_file_location("falconsproof_modeling", hf_hub_download(REPO, "falconsproof_modeling.py"))
fpm = importlib.util.module_from_spec(spec); spec.loader.exec_module(fpm)

# Config: use the repo's if complete, else the base DistilBERT config + falconsproof calibration and gate
raw = json.load(open(hf_hub_download(REPO, "config.json"), encoding="utf-8"))
report = (json.load(open(hf_hub_download(REPO, "falconsproof_report.json"), encoding="utf-8"))
          if "falconsproof_report.json" in files else {})
config = (fpm.FalconsProofConfig.from_dict(raw) if {"dim", "n_layers", "id2label"} <= raw.keys()
          else fpm.FalconsProofConfig.from_pretrained(BASE, revision=BASE_REV))
cal, gate = report.get("calibration", {}), report.get("gate", {})
for key, fallback in [("t_intent", cal.get("t_intent")), ("t_option", cal.get("t_option")), ("gate_tau", gate.get("tau")),
                      ("gate_kappa", gate.get("kappa")), ("gate_map_min", gate.get("map_min")),
                      ("option_template", None), ("max_len", None)]:
    value = raw.get(key, fallback)
    if value is not None:
        setattr(config, key, value)

# Weights: accept model_edited.safetensors and strip any key prefix the editing tool added
weights_file = next((f for f in ("model.safetensors", "model_edited.safetensors") if f in files), None) \
    or next(f for f in files if f.endswith(".safetensors"))
state = load_file(hf_hub_download(REPO, weights_file))
model = fpm.FalconsProofModel(config)
expected = set(model.state_dict())
strip = lambda k, p: k[len(p):] if k.startswith(p) else k
prefix = max(["", "model.", "module.", "falconsproof."], key=lambda p: len(expected & {strip(k, p) for k in state}))
missing, unexpected = model.load_state_dict({strip(k, prefix): v for k, v in state.items()}, strict=False)
missing = [k for k in missing if not k.endswith("position_ids")]
assert not missing, f"weights missing from {weights_file} (they would be random): {missing[:10]}"
if unexpected:
    print("ignored unexpected tensors:", unexpected[:10])
model.eval()                                     # add .to("cuda") for a GPU

# Tokenizer: the repo's if present, else the base model's (identical to what v1 was trained with)
try:
    tokenizer = AutoTokenizer.from_pretrained(REPO)
except Exception:
    tokenizer = AutoTokenizer.from_pretrained(BASE, revision=BASE_REV)
    print("tokenizer: not in the repo, using the base model's")

cfg = model.config
LABELS = [cfg.id2label[i] for i in range(cfg.num_labels)]
print(f"loaded {weights_file}: {len(LABELS)} labels | tau={cfg.gate_tau:.3f} kappa={cfg.gate_kappa:.3f} "
      f"mu={cfg.gate_map_min:.2f} t_intent={cfg.t_intent:.3f} t_option={cfg.t_option:.3f}")
# Expected for this checkpoint: tau=0.875 kappa=0.200 mu=0.50 t_intent=1.000 t_option=0.709

5.3 Decide (with question fanout)

In v1, the routing logic lives in the training notebook rather than in falconsproof_modeling.py. This self-contained function reproduces it exactly:

def softmax(x, t=1.0):
    x = np.asarray(x, dtype=np.float64) / max(float(t), 1e-6)
    e = np.exp(x - x.max())
    return e / e.sum()


@torch.no_grad()
def decide(state, questions):
    """questions: [{"question": str, "options": [str, ...]}, ...]  ->  one result per question (2 forward passes)."""
    dev = model.device
    # Pass 1: intent logits for the state and every unique option text (specialist).
    singles = list(dict.fromkeys([state] + [o for q in questions for o in q["options"]]))
    enc = tokenizer(singles, truncation=True, max_length=cfg.max_len, padding=True, return_tensors="pt")
    il = model.intent_logits(enc["input_ids"].to(dev), enc["attention_mask"].to(dev)).float().cpu().numpy()
    table = dict(zip(singles, il))
    # Pass 2: one relevance logit per (question | state, option) pair (generalist).
    firsts, seconds, spans = [], [], []
    for q in questions:
        first = cfg.option_template.format(question=q["question"], state=state)
        spans.append((len(firsts), len(firsts) + len(q["options"])))
        firsts += [first] * len(q["options"]); seconds += list(q["options"])
    enc2 = tokenizer(firsts, seconds, truncation="longest_first", max_length=cfg.max_len, padding=True, return_tensors="pt")
    scores = model.option_scores(enc2["input_ids"].to(dev), enc2["attention_mask"].to(dev)).float().cpu().numpy()

    results = []
    for q, (a, b) in zip(questions, spans):
        generalist = softmax(scores[a:b], cfg.t_option)
        native = softmax(table[state], cfg.t_intent)
        maps, map_conf = [], []
        for o in q["options"]:
            p = softmax(table[o], cfg.t_intent)
            maps.append(int(p.argmax())); map_conf.append(float(p.max()))
        specialist = softmax(np.asarray(table[state])[maps], cfg.t_intent)
        coverage = float(native[sorted(set(maps))].sum())
        collision = len(set(maps)) < len(maps)
        use_specialist = (not collision and min(map_conf) >= cfg.gate_map_min
                          and coverage >= cfg.gate_kappa and specialist.max() >= cfg.gate_tau)
        probs = specialist if use_specialist else generalist
        results.append({
            "question": q["question"],
            "choice": q["options"][int(np.argmax(probs))],
            "confidence": float(probs.max()),
            "route": "specialist" if use_specialist else "generalist",
            "probs": {o: round(float(p), 4) for o, p in zip(q["options"], probs)},
            "gate": {"conf": float(specialist.max()), "coverage": coverage, "min_map_conf": min(map_conf),
                     "collision": collision, "mapped": [LABELS[m] for m in maps]},
        })
    return results


for r in decide(
    "I was charged twice for my March invoice, please reverse the duplicate payment.",
    [{"question": "Which team should handle this message?", "options": ["billing and payment", "shipping", "delete account"]},
     {"question": "How urgent is this?", "options": ["Low", "Medium", "High"]}],
):
    print(r["question"], "->", r["choice"], f"({r['confidence']:.0%}, {r['route']})")

Tips for writing options. Options that are the exact intent names, or close paraphrases of them, give the specialist a confident mapping, which lets its gate open. Options that don't correspond to any of the 15 intents are handled by the generalist. Keep options distinct: two options that map to the same intent create a collision, and the specialist then abstains.

---

6. Architecture

                    ┌──────────────────── shared DistilBERT encoder (from Falconsai) ─────────────────────┐
 state ─────────────┤─► [CLS] ─► intent head (15 labels, frozen) ─► specialist distribution over options  │──┐
 option texts ──────┤─► [CLS] ─► intent head ─► maps each option to an intent (+ mapping confidence)      │  │
 (question | state, │                                                                                      │  │
   option) pairs ───┤─► [CLS] ─► option head (MLP → 1 logit per pair) ─► generalist distribution          │──┤
                    └──────────────────────────────────────────────────────────────────────────────────────┘  │
                                                                                                              ▼
      step-function gate:  specialist  ⇔  H(conf − τ) · H(coverage − κ) · H(min_map_conf − μ) · [no collisions]

6.1 Shared encoder

This is DistilBERT, initialised from the Falconsai checkpoint, and both heads read its [CLS] vector. During training the embeddings and the first two transformer layers are frozen, while the upper four layers adapt.

6.2 Specialist path

The pre_classifier → ReLU → dropout → classifier stack keeps exactly the parameter names of DistilBertForSequenceClassification, so the Falconsai weights load into it unchanged. It is frozen throughout training. At inference it does three jobs:

1. Classifies the state into the 15 intents.
2. Maps each option to an intent by classifying the option's own text, which also yields a mapping confidence.
3. Restricts the state's distribution to the mapped intents and renormalises it, which gives a probability per option.

It also reports coverage: the share of the state's full 15-way probability that falls on the intents behind your options.

6.3 Generalist path

Each option is encoded as a pair "{question} | {state}" [SEP] "{option}", and a two-layer MLP (768 → 768 → GELU → dropout 0.2 → 1) turns the pair's [CLS] vector into a relevance logit. A softmax over one question's options gives the distribution. Because the options are inputs rather than output classes, the label space is defined at request time.

6.4 Gate

Step	Passes when	Guards against
H(conf − τ)	The specialist's calibrated top probability is ≥ τ	Uncertain in-domain calls
H(coverage − κ)	At least κ of the specialist's belief falls on the offered intents	Inputs outside the specialist's domain
H(min_map_conf − μ)	Every option maps to an intent with confidence ≥ μ	Options the specialist can't interpret
[no collisions]	Every option maps to a different intent	Options the specialist can't tell apart

When all steps pass, the specialist answers; otherwise the generalist does. The thresholds are stored in config.json as gate_tau, gate_kappa and gate_map_min; the initial value 1.01 means closed. In this checkpoint the gate is open, with τ = 0.875, κ = 0.20 and μ = 0.50 (§9).

6.5 Why this design

Decision	Reason
One shared encoder	One artifact to deploy, and the generalist starts from support-domain representations
Frozen head plus distillation	Freezing the head alone doesn't stop encoder drift. The distillation (KD) term keeps the specialist's outputs faithful to the teacher (the Learning-without-Forgetting pattern)
Cross-encoder, listwise	Joint attention over question, state and option, trained to produce a distribution over the options
Hard gate, searched rather than learned	Step functions have no useful gradient, and only three thresholds need choosing. Every answer then comes from one named head with an auditable reason
Domain-balanced objective	Stops the larger domain from dictating the thresholds
Coverage as a signal	Detects inputs the specialist believes are about something that wasn't offered, even when its restricted confidence is high

---

7. Training data

7.1 Decision format

Every example is converted to one schema:

{"state": "…", "question": "…", "options": ["…", "…"], "answer": 1,
 "source": "clinc150", "domain": "general", "heldout": false, "intent_label": null}

Options are shuffled for every example, so the model can't learn a position shortcut.

7.2 Sources

Source	Hugging Face id	Role	Splits used	How decisions are built
Bitext customer support	bitext/Bitext-customer-support-llm-chatbot-training-dataset	In-domain (train, val, test)	carved from train	Bitext intents are mapped to Falconsai labels empirically (§7.3). Options are Falconsai labels, shown as label names or as natural paraphrases in equal measure
CLINC150 (plus)	clinc/clinc_oos	General intents (train, val, test)	train / validation / test	150 unseen intent names (plus "out of scope") as runtime-defined options
CommonsenseQA	tau/commonsense_qa	General reasoning (train, val, test)	train (train and val carved) / validation → test	5-way multiple choice
OpenBookQA (main)	allenai/openbookqa	General reasoning (train, val, test)	train / validation / test	4-way multiple choice
Banking77	PolyAI/banking77	Held out (test only)	test	Unseen intent label space
ARC-Easy	allenai/ai2_arc (ARC-Easy)	Held out (test only)	test	Unseen multiple-choice source

For intent sources, each decision has 2–6 options: the gold intent plus random distractors from the same label space. If Bitext can't be downloaded, the notebook falls back to a small synthetic in-domain set built from label paraphrases, and says so in its output.

7.3 Empirical Bitext → Falconsai label mapping

The first 64 examples of each Bitext intent are classified by the frozen Falconsai teacher. A Bitext intent is mapped to the teacher's majority label if at least 70% of those examples agree, and dropped otherwise. The 64 mapping examples per intent are then excluded from the training, validation and test pools. The resulting mapping is stored in falconsproof_report.json under bitext_intent_map.

27 Bitext intents were mapped.

Bitext intent	Falconsai label
cancel_order	cancellation
change_order	ordering
change_shipping_address	shipping
check_cancellation_fee	cancellation
check_invoice	invoicing
check_payment_methods	billing and payment
check_refund_policy	returns and refunds
complaint	complaints and feedback
contact_customer_service	speak to person
contact_human_agent	speak to person
create_account	edit account
delete_account	delete account
delivery_options	delivery information
delivery_period	delivery information
edit_account	edit account
get_invoice	invoicing
get_refund	returns and refunds
newsletter_subscription	subscription
payment_issue	billing and payment
place_order	ordering
recover_password	recover password
registration_problems	registration problems
review	complaints and feedback
set_up_shipping_address	shipping
switch_account	edit account
track_order	ordering
track_refund	returns and refunds

All 27 Bitext intents passed the 70% agreement threshold, so none were dropped. They cover 14 of the 15 Falconsai labels: Bitext has no appointment intent, so appointment is never tested in-domain. Some mappings reflect how the Falconsai classifier sees them rather than the Bitext names, for example create_account → edit account (not registration problems) and track_order → ordering (not delivery information).

7.4 Dataset licenses

Check each dataset's card before you redistribute derived data. At the time of writing, the listed licenses are:

Dataset	License
Bitext customer support	CDLA-Sharing-1.0
CLINC150	CC BY 3.0
CommonsenseQA	MIT
OpenBookQA	Apache-2.0
Banking77	CC BY 4.0
ARC	CC BY-SA 4.0

7.5 Data volumes

Volumes are set by the training preset: at most per_source_train decisions per source for training, and per_source_eval for validation and for test.

Preset	Train per source	Val / test per source	Epochs	Selected when
smoke	250	80	1	Pipeline check on a CPU
small	1,500	300	1	auto on a CPU
standard	4,000	500	2	auto on a GPU

Actual counts per split and source for this checkpoint:

Source	train	val	test
arc_easy	0	0	300
banking77	0	0	300
bitext	1500	300	300
clinc150	1500	300	300
commonsense_qa	1485	297	296
openbookqa	1498	300	300
Total	5983	1197	1796

Totals: 5,983 training, 1,197 validation and 1,796 test decisions. The test split includes 600 held-out decisions (Banking77 and ARC-Easy). Multiple-choice sources lose a few rows to malformed items.

7.6 Known data caveats

• Bitext overlap. The Falconsai training set is not public but appears closely related to Bitext. In-domain specialist accuracy on Bitext may therefore be optimistic, and Banking77 is the fairer test of intent generalisation.
• Label-space mismatch. Bitext intents that don't map cleanly to one Falconsai label (below 70% purity) are excluded, so the in-domain set covers only the unambiguous part of the taxonomy.
• Synthetic options. Paraphrased options come from a fixed phrase list, two phrasings per label, so real users' wording will vary more.

---

8. Training procedure

8.1 Initialisation and teacher check

The student is built with FalconsProofModel.from_pretrained("Falconsai/intent_classification", revision=…). A frozen copy of the original classifier is loaded as the teacher. Before training, the student's intent logits must match the teacher's to within 1e-4, and the notebook stops if they don't.

8.2 Objective

L = CE_listwise(option logits of each decision, answer)  +  λ · T² · KL( softmax(teacher/T) ‖ softmax(student/T) )

The listwise cross-entropy is computed over each decision's options, with padding masked at −1e4. The distillation term uses the states of the training batch as inputs, with λ = 1.0 and T = 2.0.

8.3 Frozen and trainable parameters

Component	Trainable
Embeddings	No
Transformer layers 1–2	No
Transformer layers 3–6	Yes (lr 3e-5)
Intent head (pre_classifier, classifier)	No
Option head	Yes (lr 5e-4)

8.4 Hyperparameters

Hyperparameter	Value
Optimiser	AdamW, weight decay 0.01
Learning rate, encoder / option head	3e-5 / 5e-4
Schedule	Linear, 6% warmup
Batch	16 decisions (flattened to all their option pairs)
Gradient clipping	1.0
Maximum sequence length	128
Options per decision	2–6 when sampled, or all choices for multiple-choice sources
Distillation λ / T	1.0 / 2.0
Precision	bf16 autocast on CUDA when supported, fp32 otherwise
Seed	42
Checkpoint selection	Best generalist validation accuracy across epochs

Monitored during training: listwise CE, distillation KL, generalist validation accuracy, and specialist agreement with the teacher on in-domain validation states.

8.5 Calibration

After training, one temperature per head is fitted by minimising NLL on the validation set, using L-BFGS over log T and clamping the result to [0.05, 20]. t_option is fitted on every validation decision; t_intent is fitted on the 15-way intent logits of in-domain validation states. Calibration changes confidence but never the argmax.

8.6 Gate optimisation (the step-function search)

For each validation decision, the search computes the specialist's calibrated view (confidence, coverage, minimum mapping confidence, collisions) and the generalist's calibrated choice. It then evaluates every combination of:

Threshold	Grid
τ (confidence)	41 values, 1.00 → 0.00
κ (coverage)	21 values, 1.00 → 0.00
μ (mapping confidence)	0.9, 0.7, 0.5, 0.3, 0.0

That is 4,305 settings in all, vectorised in NumPy. The objective is domain-balanced accuracy, the mean of in-domain accuracy and general accuracy. Grids run from high to low, so ties go to the most conservative gate, the one that routes the fewest decisions to the specialist.

---

9. Calibration and gate parameters

These values are read from this checkpoint's config.json and falconsproof_report.json.

Parameter	Value
Intent-head temperature t_intent	1.000
Option-head temperature t_option	0.709
Confidence step τ (gate_tau)	0.875
Coverage step κ (gate_kappa)	0.200
Mapping step μ (gate_map_min)	0.50
Validation objective (domain-balanced accuracy)	0.6784
Validation decisions routed to the specialist	31.4%

Base model: Falconsai/intent_classification@630d0d4668170a2a64d8d80b04d9844415bd4367.

What these values mean in practice. The specialist answers a question only when four things hold. Its calibrated top probability is at least 87.5%. At least 20% of its belief falls on the intents behind the offered options. Every option maps to an intent with at least 50% confidence. And no two options share an intent. On validation data that happened for 31.4% of decisions, and the gated model reached 67.8% domain-balanced accuracy.

Temperatures. t_option = 0.709 is below 1, so the generalist's raw scores were under-confident and calibration sharpens them. t_intent is exactly 1.000, so the specialist's probabilities are the intent head's raw softmax. An exact 1.0 can also mean the fit never moved from its starting value. Since the specialist's ECE on Bitext is low (0.025, §10), this doesn't hurt in-domain, but it is worth re-checking when retraining.

Reading the values. A τ close to 1 means the specialist answers only when it is nearly certain. A κ close to 1 means it answers only when the options cover almost all of its belief. A gate value above 1 means that condition never passes, so the specialist is effectively disabled.

---

10. Evaluation

10.1 Protocol

The test split contains all in-domain and general sources, plus the held-out sources Banking77 and ARC-Easy. Held-out data was never used for training, calibration or gate tuning. Metrics are reported per source and for all sources together:

Metric	Definition
Specialist	Accuracy of the specialist path alone, ignoring the gate
Generalist	Accuracy of the generalist path alone
falconsproof	Accuracy of the gated model, the one you deploy
Balanced	Mean per-class recall of the gated model, where the class is the position of the correct option
Perturbed	Accuracy after the options are reshuffled (seed 49), which tests position robustness
ECE	Expected calibration error of the final probabilities (10 equal-width bins)
Route	Share of decisions answered by the specialist

10.2 Results

Source	n	Specialist	Generalist	falconsproof	Balanced	Perturbed	ECE	Route → specialist
bitext	300	0.950	0.843	0.973	0.975	0.973	0.025	87.7%
clinc150	300	0.453	0.450	0.463	0.414	0.463	0.118	16.3%
commonsense_qa	296	0.226	0.250	0.250	0.249	0.250	0.068	5.7%
openbookqa	300	0.253	0.280	0.267	0.267	0.267	0.110	6.7%
arc_easy (held out)	300	0.243	0.317	0.313	0.449	0.313	0.089	5.0%
banking77 (held out)	300	0.483	0.447	0.450	0.371	0.450	0.221	18.3%
All sources	1796	0.435	0.432	0.453	0.462	0.453	0.087	23.3%

Preset small, seed 42, max length 128, 1 epoch(s). Accuracies are fractions of 1.

10.3 Key findings

Source	Chance*	Best single head	falconsproof	Gain from gating	Specialist route
Bitext (in-domain)	≈ 0.29	0.950 (specialist)	0.973	+2.3 pts	87.7%
CLINC150	≈ 0.29	0.453 (specialist)	0.463	+1.0 pts	16.3%
Banking77 (held out)	≈ 0.29	0.483 (specialist)	0.450	−3.3 pts	18.3%
CommonsenseQA	0.20	0.250 (generalist)	0.250	0.0 pts	5.7%
OpenBookQA	0.25	0.280 (generalist)	0.267	−1.3 pts	6.7%
ARC-Easy (held out)	≈ 0.25	0.317 (generalist)	0.313	−0.3 pts	5.0%
All sources		0.435 (specialist)	0.453	+1.8 pts	23.3%

*Chance is the expected accuracy of a random pick. Intent decisions have 2–6 options chosen uniformly at random (mean of 1/k ≈ 0.29); CommonsenseQA has 5 options and OpenBookQA 4.

1. The gate does its job in the home domain. On Bitext the gated model (97.3%) beats the specialist alone (95.0%) and the generalist alone (84.3%). The gate sends the 87.7% of decisions the specialist is sure about to the specialist, and lets the generalist handle the rest.
2. Combining helps overall. Across all sources the gated model is 1.8 points above the best single head.
3. It under-routes on unfamiliar banking intents. On held-out Banking77 the specialist alone would have scored 48.3%, but the gate sent it only 18.3% of decisions, giving 45.0%. Re-tuning the gate on banking-style traffic should recover this (see the developer guide's tune_gate).
4. The generalist doesn't reason. On CommonsenseQA, OpenBookQA and ARC-Easy it scores at or barely above chance. The small preset (1 epoch, 1,500 decisions per source) and a 67M-parameter encoder were not enough to learn multiple-choice QA. Treat any use outside support routing as unproven.
5. No position bias. Perturbed equals falconsproof on every source. This is expected by design: each option is scored independently, so reordering the options can't change the choice.
6. Calibration is good in-domain and poor on Banking77. ECE is 0.025 on Bitext but 0.221 on Banking77, where confidence overstates accuracy. Don't use a single confidence threshold across domains without checking it.

10.4 How to read the results

• Bitext. Compare specialist with falconsproof. The gate should keep most of the specialist's in-domain accuracy.
• CLINC150, CommonsenseQA and OpenBookQA. falconsproof should track the generalist, and route should be low.
• Banking77 and ARC-Easy (held out). These are the honest zero-shot numbers.
• Perturbed close to falconsproof means the model reads the options rather than their positions.
• A low ECE means that confidence thresholds, such as a deferral cut-off, behave as expected.

10.5 Reproducing the evaluation

Run falconsproof.ipynb top to bottom with preset = "small" and seed 42. The notebook writes the table above to falconsproof_report.json. Results vary with the preset, the hardware (bf16 versus fp32), and dataset versions on the Hub.

---

11. Behavioural test suite

The training notebook runs these checks on the model after reloading it from disk. The notebook's printed output records whether each one passed for a given training run:

#	Check	Pass criterion
1	Probabilities are valid	Each question's probabilities sum to 1 (±1e-6)
2	Determinism	Two identical calls return identical probabilities
3	Save/load fidelity	Option and intent logits match to <1e-4; temperatures, gate and template round-trip; test metrics identical after reload
4	Fanout consistency	Asking questions together equals asking them separately (±1e-3)
5	Order robustness	The chosen option text is unchanged after shuffling, on ≥ 90% of 200 test decisions
6	In-domain routing	"I forgot my password and can't sign in" with intent-name options routes to the specialist
7	Off-domain routing	A deployment question ("canary shows a 3% error increase") routes to the generalist
8	Specialist preserved	The intent head agrees with the original Falconsai model on ≥ 95% of in-domain test states

A failing check is treated as a finding about the checkpoint, typically an undertrained preset, rather than as a crash.

---

12. Bias, risks and limitations

Technical limitations

• Closed world. The model always picks one of the options, even when none is right.
• 128-token window. Long messages are truncated, which can remove the deciding detail.
• Small backbone, light training. A 67M-parameter DistilBERT trained with the small preset: multiple-choice accuracy on science and commonsense questions is at or near chance (25–31%).
• Fixed specialist vocabulary. The specialist knows only 15 support intents. Everything else relies on the generalist.
• Calibration is distribution-dependent. Temperatures were fitted on the validation mix. On held-out Banking77 the calibration error is 0.221 (versus 0.025 in-domain), so confidence can be badly overstated on unfamiliar traffic.
• Mapping errors. The specialist maps options by classifying their text. Unusual phrasings can be mapped to the wrong intent with high confidence. The μ threshold and the collision check reduce this risk but don't remove it.

Sociotechnical risks

• Language and dialect. The training data is English and mostly templated or crowd-sourced. Performance on dialects, code-switching, informal spelling or non-native phrasing is untested and likely lower.
• Inherited bias. Biases in the base model's pre-training data, in Bitext's synthetic customer language, and in the QA datasets can carry over into the decisions.
• Automation risk. Routing customers automatically can systematically misroute groups whose phrasing differs from the training data, for example people with disabilities who use assistive phrasing, or non-native speakers.
• Over-trust in confidence. A calibrated confidence is an average property. It doesn't guarantee that any single decision is right.

---

13. Recommendations

• Evaluate on your own traffic (50–500 labelled decisions) before deploying, and compare the results with §10.
• Use a deferral policy. Act on specialist-routed answers, or on generalist answers above a confidence threshold chosen from your evaluation. Send everything else to a human.
• Keep humans in the loop for irreversible or consequential actions.
• Log the route, confidence and model revision with every decision, and review the misroutes regularly.
• Monitor drift. Re-check calibration (ECE) whenever your traffic, product lines or channels change.
• Retrain with the standard preset on a GPU (4,000 decisions per source, 2 epochs) before relying on the generalist, and re-tune the gate on your own traffic if it resembles Banking77 more than Bitext.
• Prefer falconsproof v2.0 for new work: it has a longer context, a stronger generalist, and code and reasoning specialists.

---

14. Environmental impact

Not yet filled. Record the hardware, training time and region of your training run here. Estimate emissions with the ML CO2 Impact calculator.

Hardware	—
Training time	—
Cloud provider / region	—
Estimated emissions	—

This checkpoint was trained with the small preset (about 6k training decisions, 1 epoch), which the notebook's auto setting selects on a CPU. As a guide, v1 is small: the standard preset runs about 10k decisions for 2 epochs on a 67M-parameter model, which takes minutes to tens of minutes on a single modern GPU. Inference needs no GPU.

---

15. Technical specifications

Objective	Listwise CE + KD (T = 2)
Pooling	[CLS] token of the final layer
Pair format	"{question} | {state}" [SEP] "{option}", truncated longest_first to 128 tokens
Inference passes	2 per call: (1) state and unique option texts through the intent head; (2) all (question, option) pairs through the option head
Software	Python ≥ 3.9 (3.11 recommended), PyTorch ≥ 2.1, transformers ≥ 4.44 and < 5, safetensors, NumPy. This checkpoint's config was saved with transformers 4.57.6
Serialisation	model_edited.safetensors (float32, also as model_edited.F32.gguf) plus config.json (model_type: falconsproof, architectures: ["FalconsProofModel"])
Determinism	Bit-for-bit repeatable in eval mode with the same hardware and library versions

Config fields added on top of DistilBertConfig:

Field	Meaning	Initial	This checkpoint
t_intent	Intent-head temperature	1.0	1.000
t_option	Option-head temperature	1.0	0.709
gate_tau	Confidence step τ	1.01 (closed)	0.875
gate_kappa	Coverage step κ	1.01 (closed)	0.200
gate_map_min	Mapping-confidence step μ	1.01 (closed)	0.50
option_template	First-segment template for pairs	"{question} | {state}"	same
max_len	Tokenizer truncation length	128	128

The other DistilBERT settings in config.json are unchanged from the base model: 6 layers, 12 heads, dim 768, hidden dim 3072, vocabulary 30,522, 512 positions, dropout 0.1, and classifier dropout 0.2.

---

16. Files in this repository

File	Contents
config.json	Architecture, 15 labels, temperatures, gate thresholds
model_edited.safetensors	All weights (float32, 270 MB): encoder, intent head, option head
model_edited.F32.gguf	The same weights in GGUF format
falconsproof_modeling.py	FalconsProofConfig and FalconsProofModel
falconsproof_report.json	Base-model revision, labels, calibration, gate search, Bitext mapping, training config, data counts, test results
lineage.intoto.jsonl, manifest.json, surgery_log.json	Model Surgeon lineage attestation, manifest and operation log
falconsai_surgeon_package_*.zip	The complete Model Surgeon package
README.md	This model card

The repository has no model.safetensors and no tokenizer files, so plain from_pretrained("Falconsai/proof_V_1") fails. The loader in §5.2 handles both. The tokenizer is the base model's, Falconsai/intent_classification at the pinned revision.

---

17. Versioning and relation to v2

	v1 (this model)	v2.0
Backbone	DistilBERT, 128 tokens	ModernBERT-base, trained up to 512 tokens
Specialists	1 (the original Falconsai head, reused)	3 (intent, code, reasoning), each distilled from its own teacher
Gate	One, with three thresholds	One per specialist, found by coordinate-ascent search
Training data	Support intents, general intents, multiple-choice QA	Adds 11 code tasks and 7 reasoning sources
Routing code	In the notebook (reproduced in §5.3)	Shipped in falconsproof_modeling.py

Changelog

• v1.0: first release, trained with the small preset (seed 42). Frozen Falconsai specialist with distillation, listwise cross-encoder generalist, temperature scaling, and a single step-function gate found by exhaustive search.

---

18. Glossary

Term	Meaning
State	The text the decision is about
Decision	One question with its options
Specialist	The original 15-intent classifier head
Generalist	The cross-encoder option head, which handles any options
Coverage	The share of the specialist's belief that falls on the intents behind the offered options
Mapping confidence	How sure the specialist is about which intent an option means
Collision	Two options mapped to the same intent
Fanout	Several questions about one state, answered in the same two forward passes
Held out	Never seen in training, calibration or gate tuning
ECE	Expected calibration error: the average gap between confidence and accuracy

---

19. Citation and references

@misc{falconsai_proof_v1_2026,
  title        = {falconsproof v1: a specialist-generalist decision model with a step-function gate},
  author       = {{Falconsai}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Falconsai/proof_V_1}},
  note         = {Fine-tuned from Falconsai/intent_classification}
}

Methods

• Hinton, Vinyals and Dean (2015). Distilling the Knowledge in a Neural Network.
• Li and Hoiem (2016). Learning without Forgetting.
• Sanh et al. (2019). DistilBERT, a distilled version of BERT.
• Nogueira and Cho (2019). Passage Re-ranking with BERT.
• Cao et al. (2007). Learning to Rank: From Pairwise Approach to Listwise Approach.
• Yin, Hay and Roth (2019). Benchmarking Zero-shot Text Classification.
• Guo et al. (2017). On Calibration of Modern Neural Networks.
• Geifman and El-Yaniv (2017). Selective Classification for Deep Neural Networks.

Datasets

• Larson et al. (2019). An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (CLINC150).
• Casanueva et al. (2020). Efficient Intent Detection with Dual Sentence Encoders (Banking77).
• Talmor et al. (2019). CommonsenseQA.
• Mihaylov et al. (2018). Can a Suit of Armor Conduct Electricity? (OpenBookQA).
• Clark et al. (2018). Think you have Solved Question Answering? Try ARC.
• Bitext. Customer Support LLM Chatbot Training Dataset.

---

20. Model card authors and contact

Written by the Falconsai team. Please report issues, misroutes or evaluation results through the Community tab of this repository.

---

This card is generated from the surgical record itself; the package's lineage.intoto.jsonl is the signed source of truth (verify it free at the Surgeon's public verifier or with the bundled verify_attestation.py).

Architecture

• Identification: NLP · Small Language Model (SLM) (95% confidence)
• Source format: safetensors · Intended task: not declared
• config.json: the repo's config.json, edited
• Source license: apache-2.0
• Lineage chain: 2 surgeries — prior signed by ed25519:70d5116dbd76 (FALCONS.AI Model Surgeon V7.96; 4 earlier operation(s) carried) · Falconsai/proof_V_1
• Post-surgery totals: 67,556,476 parameters · 109 tensors
• Compute estimate: 5.638717 GFLOPs (comparison metric, not a measurement)

Provenance & operations

• Parents: Falconsai/proof_V_1/model_edited.safetensors
• Operations performed: load×1
• Weight merges recorded: 0
• Quantized tensors (F32→F16): 0

Surgery Log (ordered)

1. load — hub:Falconsai/proof_V_1/model_edited.safetensors (270.2 MB, safetensors)

Validation

• Tissue imaging: not run
• Structural integrity is testable offline via the packaged load_and_test.py.

Compliance note

The signed attestation + this card together document model composition, modification history, and validation evidence — the record structure technical-documentation obligations (e.g. EU AI Act Annex IV) ask for. This is evidence, not legal advice.

---

Operated with Model Surgeon — verify this package at https://surgeon.falcons.ai/verify © 2026 FALCONS.AI — Model Surgeon record format. The model weights remain their owner's.