WhissleAI/arc-v1

Arc v1

Arc v1 is a streaming audio understanding model. It listens to live audio and, every 100 ms, reports what is happening: a transcript, and on top of it tags (speaker roles, discourse acts and intents, segment types, events), names linked to your own name list, numbers with their values, speakers and their roles, and sound events (cough, breath, throat clear). Audio is always the input. Transcript words are an optional second input: Arc ships a built-in ASR decoder, and it can also read the words of an external ASR service such as Deepgram or Verbit.

• Code and Arc weights: Apache-2.0. Bundled fine-tuned Moonshine Streaming tiny: MIT (see NOTICE).
• English, 16 kHz mono (other rates are resampled). CPU or GPU, real time on a few CPU cores.
• Python package arc-audio 1.0.1 (import name arc_audio).

What's new in 1.0.1

Inference fixes only: the weights are byte-identical to 1.0.0. They change what Arc does with external ASR words (transcript="external", name_source="word"), plus a number-parsing fix that applies to every transcript source.

• Exact transcript matches are kept. If the external transcript spells a name exactly as in your list, Arc always keeps and links that mention (via: "exact"). This also covers an alias, or a surname that only one list entry has. In 1.0.0, Arc dropped about half of these mentions on external words. Two things caused it: Arc's word tagger missed some names, and the class mask dropped others (for example "London" tagged as a person). The class mask no longer drops a candidate whose name the words spell exactly. Every other mention still goes through Arc's name pointer and re-scorer.

• Number values come from the text. Arc keeps its own number kind, but parses the value from the words. This fixes values that were lost or wrong in 1.0.0:

  • "25 minutes" now has a value (1.0.0 gave none);
  • "6 p.m." is 18:00, not 06:00;
  • "3.8 seconds" now has a value;
  • "14% to 10%" and "102 to 98" are pairs, not 1410 and 10298. The quickstart's "102 to 98" is now (102, 98).

  If the text shows a number that Arc did not detect, Arc adds it from the text parser alone.

• Optional audio fallback for names (on by default). Arc's built-in transcript path runs on the same audio, alongside the external words. Where the external words gave no name record, it adds the names it links (via: "fallback"). A record from the external words is never overridden. The fallback roughly doubles the compute. strict=True turns it off, and text_layer=False restores the 1.0.0 behaviour on external words.

Measured before / after. These are 38 held-out test windows (5 minutes each, 3.2 h) with recorded Verbit live captions. All systems are scored on the same windows and gold labels:

• "Verbit text + list match" is Verbit's transcript alone, matched against the same name list with a string / phonetic rule and parsed with Arc's number parser.
• The 1.0.0 and 1.0.1 columns are both Arc v1 on the same Verbit words, run end to end.
• Brackets are 95% item-bootstrap CIs.

	Verbit text + list match	Arc v1 1.0.0 + Verbit words	1.0.1, strict	1.0.1, audio fallback (default)
names linked right	0.706 [0.66, 0.76]	0.433 [0.32, 0.53]	0.685 [0.63, 0.73]	0.745 [0.69, 0.79]
false links per link	0.621	0.520	0.588	0.609
numbers, exact value	0.862 [0.81, 0.91]	0.720 [0.68, 0.76]	0.887 [0.84, 0.93]	0.887 [0.84, 0.93]
number kind accuracy	0.826	0.929	0.929	0.929

How the 1.0.1 default compares with Verbit's text alone:

• names linked right: +0.039 [-0.02, +0.08];
• false links per link: -0.012 [-0.03, +0.01];
• numbers, exact value: +0.025 [0.00, +0.06].

The number fix also helps Arc's built-in transcript on the same windows: numbers with the exact value go from 0.549 to 0.625, and names linked right from 0.447 to 0.465.

Caveats.

• These are test windows, but the rules were chosen after looking at errors on them. Treat the numbers as indicative.
• Each domain has only 2-10 windows (sports 10, meeting 6, assistant 5, court appellate 5, news 4, talk 3, court trial 3, esports 2), so per-domain differences are not reliable.
• The gold labels for sports, news, talk, esports and court trial were made partly on other ASR transcripts. This favours methods that match transcript text, and it under-counts names that only the audio path recovers.
• The false-link rates are inflated for every system, because the gold labels are incomplete.
• The fallback column applies the fallback rule to the records of the two paths. In a live stream, Arc waits for the external words up to the word wait (12 s for Verbit) before it adds a fallback name.
• The other result tables on this card were measured with 1.0.0. The external-word rows predate these fixes.

Architecture

audio 16 kHz -> Moonshine Streaming tiny encoder (fine-tuned, 240 ms lookahead) -> pooled to 10 Hz frames
             -> Mamba-3 trunk (5 layers, d 512, carried state, domain + delay conditioning)                -> audio state h
words (optional: built-in decoder or any ASR) -> word encoder (word pieces + the audio around each word)  -> h' = h + words
             -> one tag-prompt head over a 249-tag ontology (frame, span, event, utterance, speaker read-outs)
             -> name pointer over your name list + re-scorer, numeral reader, diarization, non-speech vocal events
built-in transcript: the Moonshine decoder on the same encoder writes the text with inline name / number tags

Arc has about 24M trainable parameters (model.safetensors, 118 MB) on top of the 44M-parameter Moonshine Streaming tiny encoder / decoder. Outputs are decided at a label delay of 0.3 s by default (delay_s in 0, 0.1, 0.2, 0.3, 0.5, 0.8); wall-clock latency is the delay + the encoder's 0.24 s lookahead + compute.

Domains (pass domain=): nba, nhl, nfl, cricket, soccer, esports, news, talk, stream_chat, meeting, court_appellate, court_trial, assistant, assistant_noisy, short_clips, and unknown for anything else. A tag is only emitted in domains where it was trained; 146 tags are emitted by default and 17 experimental ones with include_experimental=True.

Install

pip install huggingface_hub
python -c "from huggingface_hub import snapshot_download; snapshot_download('WhissleAI/arc-v1', local_dir='arc-v1')"
pip install ./arc-v1                 # installs arc_audio + dependencies (torch, transformers>=5.19, safetensors, soundfile, ...)
pip install "./arc-v1[live]"         # optional: websockets, for the live Deepgram / Verbit examples
python arc-v1/tests/test_smoke.py    # CPU smoke test on the bundled synthetic clip

Arc.from_pretrained("WhissleAI/arc-v1") downloads the weights on first use; Arc.from_pretrained("arc-v1") loads a local copy.

Quickstart

1. Built-in transcript (no external service)

from arc_audio import Arc, load_audio

arc = Arc.from_pretrained("WhissleAI/arc-v1", device="cpu", threads=4)   # device="cuda" for GPU (or env ARC_DEVICE=cuda)
names = ["Marcus Delaney", {"name": "Theo Brandt", "kind": "person", "aliases": ["Brandt"]},
         {"name": "Harbor City Hawks", "kind": "team"}]                       # your name list (roster / KB), optional

# offline: one call over a file
out = arc.predict("game.wav", domain="nba", names=names, transcript="builtin")
for u in out["transcript_rewritten"]:
    print(u["start"], u["text"])
print(out["names"], out["numbers"], out["events"], out["speakers"])

# live: feed audio as it arrives (float32 mono 16 kHz, any chunk size; 1600 samples = 100 ms)
wav = load_audio("game.wav")
s = arc.stream(domain="nba", names=names, transcript="builtin")
for i in range(0, len(wav), 1600):
    new = s.feed(wav[i:i + 1600])        # only what became available: transcript, names, numbers, events, ...
final = s.flush()                         # end of stream: closes open utterances, spans and speakers

Runnable version: examples/quickstart_builtin.py (python examples/quickstart_builtin.py [audio] [domain] [cpu|cuda]). The built-in decoder writes names and numbers as inline tags; Arc links each name to your list and parses each number's value. Without a name list, names are still reported (state: "not_in_list").

2. External words from Deepgram

arc-audio has no Deepgram client of its own: call Deepgram as usual and pass its word timestamps to Arc. The accepted word format is {"text", "start", "end", "is_final"} with times in seconds on the audio clock. Use name_source="word" so that Arc tags names and numbers on the external words itself (the configuration measured below).

import io, json, os, urllib.request, soundfile as sf
from arc_audio import Arc, WordEventAdapter, load_audio

arc = Arc.from_pretrained("WhissleAI/arc-v1")
wav = load_audio("game.wav")
buf = io.BytesIO(); sf.write(buf, wav, 16000, format="WAV", subtype="PCM_16")          # send Deepgram the same audio Arc gets
req = urllib.request.Request("https://api.deepgram.com/v1/listen?model=nova-3&smart_format=true", data=buf.getvalue(),
                             headers={"Authorization": f"Token {os.environ['DEEPGRAM_API_KEY']}", "Content-Type": "audio/wav"})
dg = json.loads(urllib.request.urlopen(req).read())
words = [{"text": w["punctuated_word"], "start": w["start"], "end": w["end"], "is_final": True}
         for w in dg["results"]["channels"][0]["alternatives"][0]["words"]]
out = arc.predict(wav, domain="nba", names=names, transcript="external", words=words, name_source="word")

# live: push every Deepgram websocket message into the stream (interim and final "Results" messages are parsed as they are)
s = arc.stream(domain="nba", names=names, transcript="external", name_source="word", interim_source="deepgram")
adapter = WordEventAdapter(s)
# in the websocket receive loop:  adapter.on_message(message)
# in the audio loop (the audio you send to Deepgram):  new = s.feed(chunk)

Runnable version with the live websocket loop: examples/with_deepgram.py (DEEPGRAM_API_KEY from the environment; --live for streaming; --demo runs a local simulation without a key or network).

3. External words from Verbit

arc-audio has no Verbit client either. Book a live captioning order through Verbit's realtime API (account-specific, see Verbit's documentation), stream the audio to its websocket, and pass the caption words to Arc. Verbit's committed captions arrive a few seconds after the words; interim_source="verbit" makes Arc wait up to 12 s for them before it reads a name or number.

import json

def verbit_words(message):                         # Verbit realtime message -> Arc word events
    r = json.loads(message).get("response") or {}
    if not r.get("alternatives") or (r.get("type") != "captions" and r.get("is_final")):
        return []
    final = r.get("type") == "captions"            # captions = committed words; transcript = interim words
    out = []
    for it in r["alternatives"][0].get("items") or []:
        if it.get("kind") == "punct":
            if out:
                out[-1]["text"] += it.get("value", "")
            continue
        out.append({"text": it["value"], "start": float(it["start"]), "end": float(it["end"]), "is_final": final})
    return out

s = arc.stream(domain="nba", names=names, transcript="external", name_source="word", interim_source="verbit")
# in the websocket receive loop:  s.add_words(verbit_words(message))
# in the audio loop (the audio you send to Verbit):  new = s.feed(chunk)

Runnable version: examples/with_verbit.py (--demo simulates Verbit messages locally; --live connects to an already booked order with VERBIT_WS_URL and VERBIT_API_KEY from the environment).

More options

• transcript="none": audio only. Tags, events, spans, diarization and speakers still work; names use the audio alone and number values are not read.
• transcript="parakeet": local strong finals from NVIDIA Parakeet-TDT 0.6B v2 (pip install "./arc-v1[parakeet]", GPU recommended; downloaded on first use, not bundled).
• Roles with an external diarizer: s.add_voice_segment(voice, start_s, end_s) for each diarizer segment, then s.voice_roles() gives the role per voice (recommended when roles matter; see Results).
• Spans from your own tagger: s.add_words(words, spans=[(start, end, "ENTITY_PERSON"), ...]) with name_source="decoder".
• early_commit=True (off by default) shows some names before the final transcript arrives.

Output format

predict() returns, and stream().feed() / flush() return incrementally:

key	content
transcript / transcript_rewritten	utterances {start, end, text, words}; rewritten = linked names in your list's spelling
transcript_partial	(stream) the current partial caption, refreshed every 0.4 s
names	one per name mention: heard words, entity (your list's spelling) or null, state (linked / not_in_list / abstain), p, start / end, show (display flag)
numbers	one per number: heard, kind (score, clock, date, money, stat, ...), value (+ value2 for pairs such as scores), unit
events	{tag, t, p, window} (e.g. speaker change, score events)
spans	word-span tags {tag, start, end, p} (name classes, numerals and number kinds)
segments	frame-level exclusive groups (e.g. court phases): {group, label, start, end, p}
utterances	per utterance: speaker, acts, intents, device-directed
speakers	per speaker: role and other speaker-level tags (running posterior)
diarization	{speaker, start, end}
voice_roles	(stream, with add_voice_segment) role per external-diarizer voice
bio_events	cough / throat clear / breath events
frames	per-frame probabilities for every read-out (tags[q] = tag ids, probs[q] = frames x tags)
live, updates	(stream) the current state at the latest frame, and provisional / confirm / revise updates for names and numbers

Group members (for example speaker.role.*) are probabilities conditional on the group's gate tag. Per-tag thresholds are in config.json (thresholds); spans and events use them.

Results

All results are for this checkpoint. Systems compared in one table read the same audio windows; held-out data only (dev for the name-list baselines, test elsewhere). Labels are mostly machine-made (human transcripts and annotations where the corpora have them), so the numbers measure agreement with our labelling pipeline. Full tables: results/RESULTS.md, numbers: results/results.json.

Names: Arc v1 vs matching the transcript alone

Protocol: held-out dev split, name mentions with a session name list; every system reads the same heard words (built-in decoder, or stored commercial-ASR words). Cell = link accuracy over in-list mentions (abstain counts as wrong) / false-link rate over out-of-list mentions. Arc's per-domain name settings are cross-fitted (fitted on half of the items, scored on the other half).

mentions	words	in-list / out-of-list	transcript exact match	transcript fuzzy match	Arc v1
all domains	built-in decoder	4887 / 1421	0.46 / 0.11	0.67 / 0.38	0.64 / 0.09
sports	built-in decoder	3751 / 419	0.43 / 0.12	0.67 / 0.48	0.65 / 0.10
news / talk	built-in decoder	597 / 599	0.69 / 0.12	0.79 / 0.37	0.74 / 0.10
esports	built-in decoder	154 / 94	0.40 / 0.04	0.50 / 0.32	0.51 / 0.09
all domains	commercial ASR	4195 / 1088	0.63 / 0.13	0.87 / 0.41	0.82 / 0.09
sports	commercial ASR	3413 / 359	0.58 / 0.12	0.86 / 0.45	0.81 / 0.08
news / talk	commercial ASR	597 / 599	0.84 / 0.14	0.95 / 0.40	0.88 / 0.09
esports	commercial ASR	154 / 94	0.79 / 0.04	0.86 / 0.34	0.86 / 0.09

Exact match = the heard span contains a list name (or a surname unique in the list). Fuzzy match = the best string / phonetic match of 1-3-word sub-spans with similarity >= .85. Fuzzy matching links as many or slightly more in-list names (up to +.07), but it also links .32-.48 of the out-of-list names to a wrong list entry, against .08-.10 for Arc. Exact matching keeps false links low but misses many names (misheard or partial spellings).

Streaming evaluation on held-out test audio (built-in transcript)

Protocol: the release Stream replayed on 117 held-out test windows (300 s each, 9.75 h; sports, esports, news, talk, stream chat, meetings, court, voice assistant, cough / breath recordings), 100 ms feeds, label delay 0.3 s, built-in transcript with finals 1 s after each utterance; names linked against each window's name list. Item-bootstrap 95% CIs.

metric	Arc v1 [95% CI]	n
names linked right (in-list gold mentions)	0.436 [0.39, 0.48]	1861
names linked right within 2 s of the name	0.159 [0.14, 0.18]	1861
name span F1	0.643 [0.58, 0.69]	2983
false links per link	0.385 [0.31, 0.48]	1561
out-of-list names kept as new	0.382 [0.30, 0.47]	361
numbers: exact value	0.603 [0.56, 0.65]	829
numeral span F1	0.538 [0.48, 0.59]	1114
cough F1 (cough / breath recordings)	0.693 [0.52, 0.77]	481
breath F1	0.665 [0.27, 0.74]	175
diarization error rate (DER)	0.367 [0.32, 0.40]	84 windows
speaker-change F1 (+-0.5 s)	0.270	84 windows

By content family (names linked right / false links per link / numbers exact value / DER):

family	names linked right	false links per link	numbers exact value	DER	gold names
sports	0.463	0.241	0.647	0.330	1433
esports	0.103	0.875	0.600	0.534	58
news	0.500	0.541	0.613	0.348	116
talk	0.360	0.464	0.786	0.325	114
meeting	0.429	0.333	0.591	0.443	7
court	0.246	0.596	0.484	0.383	65
assistant	0.412	0.754	0.381	-	34

DER of the emitted diarization segments (best one-to-one speaker mapping, overlap excluded) on the 84 windows with trusted speaker labels. Few gold names in meeting (7), assistant (34) and stream chat (31): wide uncertainty.

Transcript sources: Arc v1 on its built-in transcript vs Arc v1 on external ASR words

Protocol: the same test windows for both columns of a row (the windows where the external words exist): 43 windows (3.6 h; sports, esports, news, talk) with stored Deepgram file words (+0.35 s arrival), 23 windows (1.9 h; all families) with a recorded Verbit live caption session (arrival as recorded; Arc waits up to 12 s for its finals). External words use word-level spans (name_source="word"). Cell = names linked right / false links per link / numbers exact value.

windows	Arc v1 + built-in transcript	Arc v1 + external words
Deepgram (stored file words), all (43 windows)	0.473 / 0.319 / 0.649	0.576 / 0.312 / 0.791
Deepgram, sports	0.495 / 0.249 / 0.660	0.605 / 0.221 / 0.788
Deepgram, news	0.429 / 0.577 / 0.588	0.429 / 0.618 / 0.735
Deepgram, talk	0.458 / 0.333 / 0.625	0.333 / 0.308 / 0.875
Deepgram, esports	0.103 / 0.875 / 0.600	0.224 / 0.857 / 0.857
Verbit (recorded live captions), all (23 windows)	0.466 / 0.399 / 0.602	0.436 / 0.453 / 0.669

Names linked right with 95% CIs: Deepgram windows 0.473 [0.42, 0.52] built-in vs 0.576 [0.52, 0.62] with Deepgram words; Verbit windows 0.466 [0.34, 0.59] built-in vs 0.436 [0.28, 0.55] with Verbit words (CIs overlap: Verbit words do not improve names on these 23 windows). Strong external words help most with numbers. Caveat: on the Deepgram windows the test labels were partly derived from the same Deepgram transcripts, so the Deepgram column is optimistic.

Transcript word error rate of each source

Protocol: WER (%) of each transcript on the same held-out test windows (300 s each), against human references (court trial: online captions). Transcript only; Arc's tags, names and numbers are not involved. Deepgram is not listed: no Deepgram words were stored for the human-referenced windows.

domain (windows)	reference	Verbit live	Parakeet-TDT 0.6B v2 (local)	Arc v1 built-in decoder
meeting (6)	human	16.0	25.4	48.9
court, appellate (5)	human	5.3	8.2	19.2
voice assistant (5)	human	15.6	17.2	-
court, trial (3)	online captions (not verbatim)	4.8	-	15.4

The built-in decoder is small (Moonshine Streaming tiny) and far behind strong ASR on WER. Use external words when the transcript itself matters; Arc's names and numbers improve with them (table above).

Speaker roles

Protocol: held-out test windows with trusted speaker turns; per-turn role accuracy (95% CI). "External voices" = Arc's role read-out pooled per voice of an external diarizer (Stream.add_voice_segment), the recommended setup; "Arc's own speakers" = pooled over Arc's own speaker index; "gold turns" = pooled over the reference turns (upper bound).

set	windows	turns	external voices	Arc's own speakers	gold turns
court	7	191	0.92 [0.88, 0.95]	0.77 [0.73, 0.83]	0.98 [0.97, 0.99]
sports	8	146	0.77 [0.63, 0.89]	0.74 [0.68, 0.82]	0.95 [0.91, 0.98]
news	4	36	0.53 [0.07, 0.83]	0.31 [0.03, 0.67]	0.50 [0.07, 0.78]
all	19	373	0.82 [0.72, 0.89]	0.72 [0.63, 0.78]	0.92 [0.85, 0.97]

Court = appellate arguments (judge / counsel), sports = basketball, American football and ice hockey commentary, news = broadcast news (host / guest labels are unreliable; few turns).

Tags (audio only)

Protocol: the same 117 test windows, no transcript words, label delay 0.3 s; macro normalised AP per facet, nAP = (AP - prior) / (1 - prior), on labelled frames (item-bootstrap 95% CI).

facet	tags scored	macro nAP [95% CI]
speaker (roles and attributes)	16	0.389 [0.29, 0.43]
speaker.role	15	0.397 [0.29, 0.44]
discourse (acts, intents, assistant)	62	0.307 [0.27, 0.34]
discourse.act	7	0.332 [0.25, 0.41]
discourse.intent	51	0.299 [0.26, 0.34]
discourse.assistant	2	0.672 [0.60, 0.71]
entity (mention and number spans)	39	0.464 [0.43, 0.49]
entity.number (kinds)	23	0.655 [0.61, 0.70]
content	8	0.619 [0.00, 0.67]
event	2	0.452 [0.10, 0.62]
non-speech vocal (cough, breath, throat clear)	3	0.651 [0.47, 0.74]
scene	1	0.969 [0.89, 0.99]
meta (speech)	1	0.952 [0.93, 0.96]

Event timing: speaker-change events are emitted a median 0.4 s (p90 0.7 s) after the change, voice-assistant command-complete events 0.4 s (p90 0.6 s) after the command ends.

Latency

Time from the end of a spoken name / number to Arc's correct final record (test windows above, seconds, p50 / p90). Word arrival: built-in finals 1 s after each utterance, Deepgram file words +0.35 s, Verbit as recorded live.

word source	names p50 / p90	numbers p50 / p90
built-in transcript (all 117 windows)	3.3 / 9.5	3.1 / 9.1
Deepgram words (43 windows)	2.9 / 5.0	2.7 / 5.1
Verbit live captions (23 windows)	12.2 / 18.8	11.9 / 19.1

Verbit captions arrive a few seconds after the words (and Arc waits up to 12 s for them), so records come later with Verbit words.

Compute: Stream.feed() wall time per 100 ms chunk on CPU, fed in real time like a live source (4 torch threads, a shared 48-core server under load, so indicative), on the bundled 31 s clip. The 0.3 s label delay and the encoder's 0.24 s lookahead come on top.

transcript	ms per chunk p50	p90	p99	real-time factor
none (audio only)	55.4	65.1	93.4	0.56
built-in transcript (decoder on a worker thread)	83.0	96.6	119.6	0.83

Limitations

• English only, 16 kHz mono. Trained on broadcast sports commentary (basketball, ice hockey, American football, cricket, soccer), esports, news, talk shows and podcasts, meetings, appellate and trial court recordings, voice-assistant commands and gaming streams. Other audio works with domain="unknown" but with lower accuracy.
• False name links. Arc links far fewer out-of-list names than fuzzy transcript matching, but false links remain high in some settings: court, esports, stream chat and voice-assistant test windows, and, on dev, court trials and basketball with the built-in transcript. Raise the link threshold for a domain in config.json (name_policy) or show only high-confidence links where false links are costly.
• Built-in transcript quality. The bundled decoder is small and far behind strong ASR on word error rate, and it can mishear words (occasionally as profanity). Use Deepgram, Verbit or Parakeet words when the transcript itself matters.
• Calibration. Probabilities include the training prior but are not calibrated per domain; per-tag thresholds were chosen on dev data. Treat them as scores and set thresholds for your domain.
• Roles depend on segmentation. Use an external diarizer with add_voice_segment; roles over Arc's own speaker index are weaker. News host / guest roles are unreliable.
• Diarization merges voices in long sessions with many speakers (news, talk, meetings).
• Vocal events work on dedicated recordings but false-alarm in conversation; their records are emitted only in the assistant, short-clip and unknown domains.
• Evaluation labels are mostly machine-made, and some name / number labels were derived from the same kind of transcript words Arc reads, so name and number scores are optimistic.

License and attribution

Arc code and weights: Apache License 2.0 (LICENSE). The bundled fine-tuned Moonshine Streaming tiny model (moonshine-streaming-tiny/) is derived from moonshine-ai/moonshine-streaming-tiny by Useful Sensors, Inc. (dba Moonshine AI) under the MIT License; its licence and copyright notice are kept in moonshine-streaming-tiny/LICENSE. Further attributions (Mamba-3, jellyfish, the synthetic example clip) are in NOTICE. The example clip examples/example_synth.wav is synthetic speech with fictional names.

Citation

@misc{whissle2026arcv1,
  title  = {Arc v1: a streaming audio understanding model},
  author = {{WhissleAI}},
  year   = {2026},
  url    = {https://huggingface.co/WhissleAI/arc-v1}
}

Questions and issues: the repository's Community tab.