bleep-attribution — keep the wearer's speech, nothing else

Decides, every 20 ms and before anything is stored, whether speech heard by always-on glasses is the wearer's — and lets only that be kept.

The model reads acoustic features from five air microphones and a bone-conduction (BC) sensor and outputs three scores per frame (wearer, bystander, speech). A fixed, calibrated policy turns them into (wearer | bystander | unknown, start, end). Only wearer may be retained; unknown is treated as bystander, and there is no setting that keeps it. The network never produces a transcript or a voiceprint, and tests check that its inference path cannot import anything that could.

It is the frame tagger of Bleep (dilated depthwise-separable 1-D convolutions, 24 blocks, d_model 768, O(T) time, O(1) memory per streamed chunk, no KV cache), made causal with a 200 ms look-ahead, on the air/BC DSP of hybrid-voice-separation made streaming.

---

⚠️ Read this before the numbers

1. It works only as well as the BC sensor is isolated from room sound — and that was assumed, not measured. Trained with the BC channel picking up room sound at about −38 dB, (E)-base keeps

1.68%

of bystander speech. Raise that coupling to −10 dB and it keeps

57.2%

— worse than the same model with a single microphone (2.42%). This is the single most important fact on this card. A post-hoc tiny model trained with randomised coupling (−45 to −8 dB) holds

1.58%

at −10 dB, which suggests the dependence can be trained out; it is not the released model.

2. Every recording is synthetic. TTS voices, simulated shoebox rooms, a simulated five-port array (layout from a public teardown of a commercial device) and a simulated BC sensor that the commercial device does not have.

3. One boundary gate it fails, recorded rather than relaxed. Bleep's boundary test requires onset p90 |error| ≤ 600 ms and ≤ 25 % late onsets. (E)-base measures

820 ms and 0.291.

The gates were not moved. What a late onset costs in retained audio is reported directly: a mean of

14.9 ms

of each bystander segment's first 500 ms is kept.

---

What it does, measured

Test split: 240 scenes, 7 voices never heard in training or calibration. Each operating point was calibrated on 4 other voices to a leakage budget.

operating point (budget)	bystander leakage	wearer speech not kept
strict (0.5 %)	0.34%	19.9%
balanced (2 %)	1.68%	1.1%
lenient (5 %)	4.40%	0.4%

At the balanced point, against the alternatives a device maker would try first:

	bystander leakage	wearer not kept	energy / audio hour
this model, (E)-base, Neural Engine	1.68%	1.1%	1,238 J
same model, one microphone, (D)-base	2.42%	7.5%	852 J
ECAPA wearer verification, streaming, hop 500 ms	2.47%	29.3%	1,508 J (CPU)
BC level threshold, no model	1.54%	66.7%	—
always-on Whisper small + text-PII rules	70.5%	26.7%	28,285 J (CPU, its cheaper device)

Energy is the incremental whole-system energy of an Apple M2 from its battery gauge, per hour of audio processed — not a wearable's battery life. The Neural-Engine figure is for a stateless window graph that recomputes its context every call (16.0×); execution on the Neural Engine was verified with its hardware interrupt counter, and the fp16 graph changes

0.03% of retention

decisions relative to PyTorch.

Full results, every condition and every failure: RESULTS.md. Code, paper and reproduction scripts: github.com/NagaYu/bleep-attribution.

Hypotheses that failed

Written before the results (HYPOTHESES.md), judged in code:

• H15 — under a leaky BC sensor (−10 dB) the model leaks less than one microphone. Did not hold (57.2% vs2.42%).
• H13 — the bystander-onset boundary passes bleep's gates. Did not hold (above).
• H12 — bigger models leak less. Did not hold: calibration pins leakage near the budget; size shows in wearer miss instead, and not monotonically.
• H18, H18b — two onset-weighted fine-tunes fix the boundary. The second (post hoc) fixed lateness, not the p90 gate.
• H18c (post hoc, held as stated) — the same fix on this model, E-base-onset2, brings the late-onset fraction to0.151 (gate passes) while the p90 gate still fails (840 ms); leakage1.62%, wearer miss1.4%. It is available as checkpoints/E-base-onset2.pt; the measured exports and every other number on this card are for E-base.

Model

architecture	bleep frame tagger, causal convolutions, 200 ms look-ahead
parameters	87,101,571
MACs per output frame	86,726,400
input	136 causal features per 10 ms: air log-mel and band SNRs, BC log-mel and SNRs, bone-adoption weights, air/BC coherence and delay, GCC-PHAT on 6 microphone pairs, steered response, inter-microphone level differences
output	3 sigmoid channels at 50 Hz (wearer, bystander, speech) → fail-closed policy
training	5,000 steps, effective batch 16, plateau rule fired (converged)
chunked = whole-file	bit-identical on the CPU exact path (tests/test_exact_inference.py)

Files

file	what it is
checkpoints/E-base.pt	PyTorch weights + fixed feature normalisation
exports/E-base.window_h32.mlpackage	Core ML, Neural Engine, stateless window graph (the measured one)
exports/E-base.stream_h32.mlpackage	Core ML, CPU, stateful streaming graph (every frame computed once)
exports/E-base.window.onnx, exports/E-base.stream.onnx	ONNX, dynamic-length and streaming
exports/E-base.operating_points.json	the three calibrated policies
exports/E-base.export_report.json	conversion checks and Neural-Engine verification

Usage

import json, numpy as np
from attribution.dsp.features import extract, feature_columns
from attribution.model import load_attribution, predict_scores
from attribution.policy import RetentionPolicy, frame_labels, retain_mask, to_spans

model, norm, _ = load_attribution("checkpoints/E-base.pt")
pol = RetentionPolicy.from_dict(json.load(open("exports/E-base.operating_points.json"))["balanced"])

# audio: (6, n) float32 at 16 kHz -- 5 air microphones, then the BC channel
x = norm.apply(extract(audio[:5], audio[5]), feature_columns(model.cfg.feature_set))
scores = predict_scores(model, x)        # (T, 3) at 50 Hz
labels = frame_labels(scores, pol)
keep = retain_mask(labels, pol)          # True only on wearer frames
spans = to_spans(labels, scores)         # (wearer | bystander | unknown, start, end)

A runnable version is examples/attribute_scene.py.

Intended use, and what this is not

For research on keeping bystander speech out of what always-on wearables store. It is not a privacy guarantee, not evaluated on real people or real hardware, not a speaker-identification system (it cannot tell who a bystander is), and not a substitute for the legal analysis a deployment needs. Audio is held for 200 ms plus one hop in a volatile buffer before the decision; "not retained" means not stored and not passed on, not "never processed".

Reproducing

bash bench/reproduce.sh runs every step: corpus, training, baselines, exports, energy, figures, claim verification, tests.

Licence

Apache-2.0. The vendored DSP in attribution/dsp/ is MIT. Speech rendered with the macOS voices is not redistributable and is not published.