BISKY

Reference implementation of the BISKY coordination mechanism, in two variants. Both use the same routing signature, the same dynamic bipartite agent-channel graph and the same TIES merge operator. They differ only in what crosses a channel.

Variant	Policy input	Shared object	File
Belief merging	x_{i,t} = (o_{i,t}, b_{i,t}, c_{i,t})	belief adapter update Δθ_i	bisky/variant_belief.py
Policy update merging	x_{i,t} = (o_{i,t}, c_{i,t})	PPO update g_i	bisky/variant_policy.py

The environment, evaluation protocol, baselines, datasets and figures are described in the appendix and are not part of this submission.

Install

pip install -r requirements.txt

Python 3.10 or newer.

Use

Supply any environment implementing the interface in bisky/env_api.py, then build a trainer.

from bisky import PolicyMergeTrainer, PPOConfig

cfg = PPOConfig(rollout_episodes=4, epochs=6)
trainer = PolicyMergeTrainer(env, cfg=cfg, sig_obs_idx=(...), seed=0)
trainer.train(rounds=200)

Use BeliefMergeTrainer for the belief variant, adding target_fn and belief_target_dim.

Two settings are environment specific. sig_obs_idx selects the observation channels that enter the routing signature; name the channels describing the agent's own operating regime and leave out channels that are identical across agents, since those only inflate every cosine. target_fn returns the local quantities the belief model forecasts, and every entry must be something the agent observes about itself.

BISKY also runs unchanged on the SustainGym multi-agent environments. Those expose PettingZoo-style dicts, so a thin wrapper is needed to stack the per-agent observations into one [N, obs_dim] array and map actions from [0, 1] onto the environment range. Nothing inside bisky/ changes.

Routing

Each agent forms a unit signature s_{i,t} from its own recent history and connects to one similar channel and one complementary channel.

c_sim  = argmax_c cos(s_{i,t}, p_{c,t})
c_comp = argmin_c cos(s_{i,t}, p_{c,t})

A best similarity below the birth threshold creates a new channel from the agent's own signature. Switching requires beating the current channel by a fixed margin. Empty channels are deleted. Complementary members read the channel context and never contribute updates.

TIES merge

Each update is trimmed to its top k entries by magnitude, one dominant direction is elected per parameter by weighted signed mass, and only the contributions agreeing with that direction are averaged. A plain average lets opposing updates cancel; the disjoint merge does not.

updates  [+1 +1  0]      mean  [+1.00  −0.33  0]
         [+1 −1  0]      TIES  [+1.00  −1.00  0]
         [+1 −1  0]

Contribution weights combine local experience and local forecast error, w_i = n_i / (1 + L_i).

Variant 1, belief merging

Each agent trains a local self-supervised forecaster on its own state-action history. Only the residual bottleneck adapter is shared, and every adapter starts from the same fixed initialization, so member updates are comparable.

Δθ_i = θ_i − θ_init
L_i  = (1 / N_i^val) Σ_t || f_i(H_{i,t}) − y_{i,t} ||²

The channel TIES-merges the member updates weighted by w_i. Each member scores the result on its own held-out episodes and accepts it only when its forecast error does not increase.

θ_new = θ_local + α_i Δθ_merge

α_i rises when channel updates have been useful and falls when they have not. The channel also pools member forecasts, and each agent blends the channel forecast into its own with a reliability weight ρ_i.

b_{i,t} = (1 − ρ_i) b_loc + ρ_i b_ch

PPO stays local in this variant and no policy information crosses a channel.

Variant 2, policy update merging

No belief model. Each similar channel holds a channel policy θ_c shared by all of its members. Every member computes a PPO gradient on a minibatch of its own rollout buffer, the channel merges the member updates, and one optimizer step is applied.

θ_c ← θ_c + Opt( (1 / |M_c|) Σ_{i ∈ M_c} g_i )

Merging happens before any member update is applied. Applying first and averaging afterwards cancels conflicting updates, and merge_before_apply=False selects that ordering for comparison. The merge runs at the granularity of single optimizer steps on small member minibatches.

A re-routed agent adopts the policy of its new channel, so a joining agent inherits a mature policy instead of starting from scratch. Merging is scoped by the graph, so agents in different behavioral regimes never average against each other.

Privacy

A channel receives an update vector, a unit signature, a sample count and a validation loss. It never sees an observation, an action, a reward, a trajectory or any global state. There is no centralized critic. Channel information handed back to agents is delayed by one routing round, so execution stays decentralized.

Files

bisky/signature.py      the routing signature s_{i,t}
bisky/graph.py          channels, edges, birth, hysteresis, deletion, context
bisky/ties.py           trim, sign election, disjoint merge, weights w_i
bisky/belief.py         forecaster, bottleneck adapter, episodic validation
bisky/agent.py          local agent with its PPO components and acceptance gate
bisky/ppo.py            PPO, GAE, rollout buffer, policy bundle
bisky/networks.py       actor, critic, normalizer, flat vector plumbing
bisky/trainer.py        the online training loop
bisky/variant_belief.py belief adapter merging
bisky/variant_policy.py policy update merging
bisky/env_api.py        the environment interface BISKY requires

Options

Name	Default	Meaning
birth_threshold	0.90	similarity below which an agent starts a new channel
switch_margin	0.05	margin an alternative channel must beat
trim_k	0.2	fraction of entries TIES keeps per update
merge_op	ties or mean	merge operator
warmup_rounds	3	rounds before the first routing round
routing_every	1	routing interval in rounds
use_complementary	True	enable read-only complementary edges
val_every	5	every n-th episode held out for belief validation
belief_window	16	forecast input window
belief_horizon	8	forecast horizon
adapter_rank	8	bottleneck rank of the shared adapter