exp1-extractive — Arabic tool-calling label-semantic BiEncoder

A label-semantic bi-encoder for Arabic function/tool calling. Given a user utterance it does two things with a single encoder pass:

1. Tool selection — matches the whole query against candidate tool descriptions (InfoNCE-style similarity; a none pseudo-tool wins when no tool is needed).
2. Parameter span extraction — scores every query word against each parameter's description and extracts the argument value as a surface span (word-level label-semantic BIO with exclusive word assignment).

It is the extractive component of the AISA Arabic Function-Calling shared task — an alternative to a generative tool-caller that is small, fast, and interpretable.

• Base model: intfloat/multilingual-e5-base (278M, XLM-RoBERTa)
• Encoding: query prefixed with "query: ", tool/param descriptions with "passage: "; embeddings L2-normalised.
• Precision: trained bf16.

Training data

Trained on label_semantics_makeen_full.jsonl (74,759 examples: 47,679 positive / 27,080 negative), derived from the hint-free makeen-tool-calling-v0.1 ShareGPT tool-calling corpus and converted to the unified label-semantic format (query / words / tool_label / tool_candidates / param_golds). ~95.8% of gold arguments are aligned to a surface span (92.5% by the rule-based matcher, +3.3% by a secondary span-recovery pass covering number/date/currency/enum maps, list-region spans, and fuzzy matching).

epochs	3
batch size	32 (× 2 grad-accum)
learning rate	2e-5
temperature	0.05
max length	128
validation	label_semantics_dev_full.jsonl (545 rows)

Results (dev, 545 rows)

Evaluated with the shared task's aisa-label-semantics-eval (exclusive word assignment):

metric	value
Tool accuracy	0.9761 (532/545)
Param precision	0.9328
Param recall	0.9471
Param F1	0.9399
Span exact-match	0.8860 (878/991)

Note: dev has 4 candidate tools per example (not full retrieval). Compared with the in-distribution v1 checkpoint (trained on the narrow AISA train split), this model has higher tool accuracy (+1.1 pts) with param-F1 / span-EM within ~1 pt, while training on the far broader makeen tool vocabulary — so it is expected to generalise better to unseen tools.

Intended use & how to load

This is a custom bi-encoder (an e5 encoder + a task head/config), not a plain transformers model. Loading requires the shared task codebase (which defines tool_calling.training.label_semantics):

from tool_calling.training.label_semantics.model import BiEncoder

model = BiEncoder.load("NAB11/exp1-extractive")   # or a local snapshot path
pred = model.predict(query, tool_candidates, param_labels)

Or via the CLI once the repo is installed (pip install -e .):

aisa-label-semantics-infer \
  --input  your_queries.jsonl \
  --output preds.jsonl \
  --model  NAB11/exp1-extractive

The checkpoint contains the e5 encoder (config.json, model.safetensors), its tokenizer, and training_config.pt (the bi-encoder Config: prefixes, temperature, max_span_words). training_config.pt is a pickled dataclass, so the tool_calling package must be importable when loading.

Limitations

• Extracts surface-span arguments; values that are computed / translated / inferred with no surface form in the query (e.g. ISO codes, default counts, transliterated tickers) are out of scope for the span head and are better handled by a generative model or post-processing.
• Tuned for Arabic (MSA + Gulf/Egyptian/Levantine/Maghrebi dialects) tool-calling utterances; behaviour on other domains is untested.
• Tool selection is scored against the candidate tools you provide.

Citation

Part of the AISA Arabic Function-Calling shared task. Base encoder: intfloat/multilingual-e5-base.