DeciCo-RSI

Decision-Coordinated Recursive Self-Improvement

Learn where to improve, what assistance to request, and what to retain.

Research prototype · v0.1 · BFCL single-turn and multi-turn tool use

Overview · Method · Three operators · Early results · Quick start

Overview

An agent's failure does not always call for more training. It may need better trajectories, a better execution harness, or a different training recipe. These interventions have different prerequisites and outcomes, and their value changes as the system improves. DeciCo-RSI learns to coordinate improvement attempts from evidence rather than relying only on a fixed schedule.

Two ideas connect the framework: contextual Bayesian learning decides where to search and whom to ask; genetic-algorithm-inspired candidate evolution turns those decisions into tested updates through variation, selection, and inheritance. The improved Student then participates in subsequent execution and proposal generation.

Beliefs guide exploration. Measurements govern inheritance.

Method

One system state, two levels of decisions

At round $t$, the system is represented by

$$S_t=(M_t,H_t,D_t,X_t)\mathrm{.}S\_t=(M\_t,H\_t,D\_t,X\_t).$$

$M_t$ is the Student model, including its accepted adapter; $H_t$ is its execution harness; $D_t$ is verified training data; and $X_t$ is the available decision context: failures, local outcomes, attempt history, state maturity, and budget. Accepted artifacts change $M$, $H$, and $D$; observed evidence refreshes $X$. Protected evaluation answers and future outcomes are excluded from decision context.

The outer scheduler selects one feasible operator: Data, Harness, or Model Improvement. Inner routing selects an eligible help strategy or proposer within that operator. Rules first enforce legality, budgets, and attempt limits; Thompson Sampling (TS) compares the remaining alternatives. Forced initial actions and a sole legal action run directly.

Bayesian learning: value and uncertainty from feedback

Each decision family uses Bayesian linear regression with action-specific parameter blocks:

$$r=\varphi (x,a{)}^{\mathrm{\top }}w+\epsilon ,\epsilon \sim \mathcal{N}(0,{\sigma }^2),w\mid \mathcal{E}\sim \mathcal{N}(\mu ,\mathrm{\Sigma })\mathrm{.}r=\backslash phi(x,a)^\backslash top\; w+\backslash varepsilon,\backslash qquad\; \backslash varepsilon\backslash sim\backslash mathcal\; N(0,\backslash sigma^2),\backslash qquad\; w\backslash mid\backslash mathcal\; E\backslash sim\backslash mathcal\; N(\backslash mu,\backslash Sigma).$$

Here $x$ is an observed context drawn from $X_t$, $a$ an action, $\phi(x,a)$ its context features placed in that action's block, $w$ the regression weights, and $r$ the measured target. $\mathcal E$ denotes eligible feedback history; $\mu$ and $\Sigma$ are the posterior mean and covariance. The current implementation starts with $\mu=0$, $\Sigma=I$ and uses noise variance $\sigma^2=0.01$. Parameters generalize across contexts; there is no separate weight vector for every exact state–action pair.

An eligible observation changes both predicted value and uncertainty. With precision $\Lambda=\Sigma^{-1}$ and $\phi=\phi(x,a)$, the update is

$${\mathrm{\Lambda }}^{\mathrm{\prime }}=\mathrm{\Lambda }+\frac{\varphi {\varphi }^{\mathrm{\top }}}{{\sigma }^2},{\mu }^{\mathrm{\prime }}=({\mathrm{\Lambda }}^{\mathrm{\prime }}{)}^{-1}(\mathrm{\Lambda }\mu +\frac{\varphi r}{{\sigma }^2}),{\mathrm{\Sigma }}^{\mathrm{\prime }}=({\mathrm{\Lambda }}^{\mathrm{\prime }}{)}^{-1}\mathrm{.}\backslash Lambda\text{'}=\backslash Lambda+\backslash frac\{\backslash phi\backslash phi^\backslash top\}\{\backslash sigma^2\},\backslash qquad\; \backslash mu\text{'}=(\backslash Lambda\text{'})^\{-1\}\backslash left(\backslash Lambda\backslash mu+\backslash frac\{\backslash phi\; r\}\{\backslash sigma^2\}\backslash right),\backslash qquad\; \backslash Sigma\text{'}=(\backslash Lambda\text{'})^\{-1\}.$$

Primes denote the posterior after the observation. TS samples plausible weights from this posterior: useful actions can score higher as evidence accumulates, while uncertain actions remain explorable. Predictions are Gaussian scores, not bounded probabilities.

Outer selection — where to improve. Draw one weight vector and choose the largest sampled gain among legal operators:

$${\tilde{w}}_{\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{e}\mathrm{r}}\sim \mathcal{N}({\mu }_{\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{e}\mathrm{r}},{\mathrm{\Sigma }}_{\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{e}\mathrm{r}}),a^{\ast }=\arg \underset{a\in {\mathcal{A}}_{\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{e}\mathrm{r}}(x)}{\max }\varphi (x,a{)}^{\mathrm{\top }}{\tilde{w}}_{\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{e}\mathrm{r}}\mathrm{.}\backslash widetilde\; w\_\{\backslash mathrm\{outer\}\}\backslash sim\backslash mathcal\; N(\backslash mu\_\{\backslash mathrm\{outer\}\},\backslash Sigma\_\{\backslash mathrm\{outer\}\}),\backslash qquad\; a^*=\backslash arg\backslash max\_\{a\backslash in\backslash mathcal\; A\_\{\backslash mathrm\{outer\}\}(x)\}\backslash phi(x,a)^\backslash top\backslash widetilde\; w\_\{\backslash mathrm\{outer\}\}.$$

$\mathcal A_{\mathrm{outer}}(x)$ is the feasible subset of the three operators. The formal target is the accepted paired Selection gain $\Delta_{\mathrm{Gate}}$ (an accuracy fraction), or zero for an eligible explicit rejection. Local success alone does not establish outer gain.

Inner selection — how to obtain the next candidate. Independently sample the operator's local-success and formal-value posteriors, then combine their predictions:

$$\tilde{u}(x,a)=\alpha {\tilde{q}}_{\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}}(x,a)+\beta {\tilde{q}}_{\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}}(x,a),a^{\ast }=\arg \underset{a\in {\mathcal{A}}_{\mathrm{i}\mathrm{n}\mathrm{n}\mathrm{e}\mathrm{r}}(x)}{\max }\tilde{u}(x,a)\mathrm{.}\backslash widetilde\; u(x,a)=\backslash alpha\backslash widetilde\; q\_\{\backslash mathrm\{local\}\}(x,a)+\backslash beta\backslash widetilde\; q\_\{\backslash mathrm\{formal\}\}(x,a),\backslash qquad\; a^*=\backslash arg\backslash max\_\{a\backslash in\backslash mathcal\; A\_\{\backslash mathrm\{inner\}\}(x)\}\backslash widetilde\; u(x,a).$$

Each $\widetilde q=\phi(x,a)^\top\widetilde w$ is a sampled prediction from its respective posterior; $\widetilde u$ is the combined score. $\mathcal A_{\mathrm{inner}}(x)$ contains eligible help sources or proposers. The current weights are $\alpha=0.05$ for local progress and $\beta=2$ for formal gain. TS chooses the route; the chosen Student or Teacher generates the actual Harness or Recipe.

Candidate evolution: vary, evaluate, inherit

The evolutionary idea operates over concrete system artifacts. A parent provides the current model, Harness, and data; feedback-guided proposals create alternatives; local checks screen them; formal comparisons decide which update survives. Accepted artifacts become the basis for later improvement.

The v0.1 search is bounded and genetic-algorithm-inspired: it implements candidate variation, selective retention, and inheritance, not explicit genome encoding or crossover. Harness and Model candidates compete against the same frozen parent, with at most one inherited winner per round. Data accumulates a verified trajectory pool and submits it together with the trained model. Bayesian routing allocates attempts within this search rather than replacing measured candidate evaluation.

Three improvement operators

Data Improvement — evolve useful training evidence. Construct tasks from observed failures and let the Student attempt them first. If assistance is needed, TS selects among legal Student retries, temporary Harness help, and Teacher solutions. Verify and deduplicate actual trajectories, freeze a training-qualified pool, then train with the formal Harness and a fixed recipe. The gate accepts Data + Model together; temporary Harness assistance stays local.

Harness Improvement — evolve the execution scaffold. The Student proposes a complete initial Harness. Follow-up proposals come from Student revision or Teacher authorship selected by TS, informed by target/control task feedback. The default search tests up to three local candidates and formally compares up to two qualified alternatives against one frozen parent. At most one Harness is inherited; no Data-generated diagnostic-task branch is invoked.

Model Improvement — evolve the training recipe. Hold Data and Harness fixed. The Student proposes the first recipe; TS routes subsequent proposer using training-health and task feedback. The default search allows two training candidates, including exploration after initial local success. Each starts from the same latest accepted model, and at most one Model passes into the formal state. Healthy training is a prerequisite, not proof of capability gain.

Two feedback signals, distinct credit

Local feedback asks whether a completed logical action made verified progress; formal feedback asks whether its attributed contribution produced an inherited gain. Separate regressors preserve both facts: a locally successful revision rejected by the gate can retain local target 1 and formal target 0. Missing measurements and infrastructure errors are not silently converted into performance failures.

For Data, one help action yields one local observation regardless of sample count; formal feedback uses a weighted joint feature vector for contributions to the consumed pool, rather than treating every sample as an independent gain observation. This is additive contribution modeling, not identified causal attribution. For Harness and Model, eligible formal feedback follows the candidate's submitting proposer.

Beta counts summarize success/failure by node and failure group; they are not sampled by contextual TS. Audit and sealed-test measurements remain outside the learning loop. The gate governs inheritance, but repeated adaptation to Selection can still overfit it; independent evaluation is needed to establish transfer.

Eight-category BFCL dataset

The default experiment now includes simple_python, multiple, parallel, parallel_multiple, multi_turn_base, multi_turn_miss_func, multi_turn_miss_param, and multi_turn_long_context. All eight categories enter Data/Harness/Model improvement and protected evaluation, with 1/8 weight per category in training and reported scores.

Multi-turn episodes maintain simulated tool state and dialogue across turns. Missing tools are released on their scheduled turn; missing parameters require abstention or clarification. Success requires the pinned BFCL state and tool-result checks across the entire episode, rather than matching one static call array. Verified assistant steps become training targets with their actual preceding dialogue; multiple steps still count as one solver-action observation.

The snapshot contains 985 static + 800 multi-turn tasks. One upstream missing-function case has a contradictory tool-release schedule and is explicitly excluded, leaving 1,784 eligible tasks. Related variants stay in one split. See the multi-turn integration guide for counts, the exclusion, execution/proof contracts, and validation limits.

Early results

One historical four-static-category, 12-round BFCL run accepted 4 updates: two Harness updates and two Data-driven training updates.

Retained-system metric	Initial	Final	Change
Selection score	84.375%	91.875%	+7.500 pp
Independent Audit score	90.625%	94.583%	+3.958 pp

Both metrics report source-balanced task-group accuracy, expressed as percentages; Selection score and Independent Audit score are shorthand for these weighted metrics. Selection is the stored single-evaluation score; Audit is a fresh three-repeat mean. Gate decisions use paired repeated evidence, so an accepted update need not increase the plotted single-evaluation Selection score. Audit can fluctuate even when the retained state is unchanged.

These results demonstrate the improvement loop, not yet the advantage of Bayesian routing. Random, fixed-order, and LLM-based routing comparisons across multiple seeds remain future work. These historical measurements are not a new benchmark of this release; the release archive does not include the original adapters or full run artifacts. See the result snapshot for round-level values and provenance.

Quick start

For the reference Python 3.12 CPU environment on Linux/macOS/WSL, run from the repository root:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.lock
python verify_bfcl_release.py
python smoke_test.py

These are offline checks, not a production GPU run. For model requirements, private execution copies, services, multi-seed runs, resume, and result inspection, follow the installation and experiment guide. The CPU lock does not define a production CUDA/vLLM environment.

License

Project code is licensed under Apache-2.0. See NOTICE, data provenance, and the BFCL AST checker license and multi-turn simulator license for third-party materials. Model weights are obtained separately under their applicable licenses.