NagaYu/echocache-matcher

EchoCache Matcher

The matching core of EchoCache as a single numpy-only file. It decides whether a cached LLM answer may be reused for a new prompt — the slot a sentence-embedding model normally fills in a semantic cache.

⚠️ This repository contains no trained weights. It is a deterministic, rule-based matcher: hashed character n-grams, a 64-bit SimHash, a secret screen and a five-check semantic guard. Nothing is learned, nothing is downloaded at runtime, and identical input always produces identical output on every machine. It is published as a model repository because of the role it plays, not because it is a neural network. If you expected weights, that expectation is wrong here and the rest of this card explains why that is the point.

Why a rule-based matcher

A semantic cache has one failure mode that matters: serving an answer written for a different question. Embedding similarity of 0.95 means "these strings look alike", not "these have the same answer". Can I cancel my subscription? and Can I **not** cancel my subscription? are near-identical to any similarity function and have opposite answers.

This matcher therefore separates the two jobs:

1. Similarity decides which entries are candidates — hashed character 3-grams, L2-normalized, cosine. No tokenizer, no word boundaries, so Japanese, Chinese and Thai behave exactly like English.
2. MismatchGuard decides whether a candidate may actually be served. Five checks veto a reuse even above the threshold: negation flip, quantity/unit, proper noun or model number, time anchor, question type.

Every rejection comes back with a machine-readable reason (numeric_mismatch:q={2024},c={2025}), so the decision is auditable.

Measured results

Scored on NagaYu/echocache-guard-benchmark: 131 prompt pairs — 41 a cache may reuse, 90 high-similarity pairs it must not.

threshold	guard	accuracy	reuse recall	wrong reuse
0.92 (default)	on	97.7%	92.7%	0 / 90
0.92	off	93.1%	92.7%	6 / 90
0.86	on	97.7%	95.1%	1 / 90
0.80	on	99.2%	100%	1 / 90
0.80	off	77.9%	100%	29 / 90
0.70	off	69.5%	100%	40 / 90

The guard is what makes a lower threshold survivable: at 0.80 it turns a 32% wrong-answer rate into 1%. Reproduce with python3 evaluate.py in the dataset repo.

Speed (developer laptop, CPU only): ~0.2 ms to featurize a prompt, ~0.3 ms for a full miss against 5,000 entries.

Usage

from huggingface_hub import hf_hub_download
import importlib.util, sys

path = hf_hub_download("NagaYu/echocache-matcher", "echocache_matcher.py")
spec = importlib.util.spec_from_file_location("echocache_matcher", path)
matcher = importlib.util.module_from_spec(spec)
sys.modules["echocache_matcher"] = matcher
spec.loader.exec_module(matcher)

matcher.match("Hi team, how do I reset my password? Thanks!", "How do I reset my password?")
# {'reuse': True, 'stage': 'exact', 'similarity': 1.0, 'guard_reasons': [], ...}

matcher.match("Can I cancel my subscription?", "Can I not cancel my subscription?", 0.5)
# {'reuse': False, 'similarity': 0.9101, 'guard_reasons': ['negation_mismatch:q=True,c=False'], ...}

Or just download the file and drop it into your project — it imports numpy and the standard library, nothing else.

Building your own cache with it

import numpy as np

index = {}                                     # exact key -> (prompt, answer, vector)

def lookup(prompt, threshold=0.92):
    key = matcher.exact_key(prompt)
    if key in index:                           # stage 1: normalized exact match
        return index[key][1]
    qv = matcher.char_ngram_hash(matcher.normalize(prompt), 3, 4096)
    best, best_sim = None, 0.0
    for cached_prompt, answer, vec in index.values():
        sim = float(np.dot(qv, vec))           # stage 2/3: cosine over candidates
        if sim > best_sim:
            best, best_sim = (cached_prompt, answer), sim
    if best and best_sim >= threshold:
        safe, reasons = matcher.is_semantically_safe(prompt, best[0])
        if safe:                               # MismatchGuard has the final say
            return best[1]
    return None

def store(prompt, answer):
    safe, reason = matcher.should_cache(prompt, answer)   # never cache a secret
    if not safe:
        return reason
    norm = matcher.normalize(prompt)
    index[matcher.exact_key(prompt)] = (prompt, answer, matcher.char_ngram_hash(norm, 3, 4096))
    return "ok"

The full production version — per-tenant partitions, LRU eviction, TTL, a SimHash candidate band, observability and a threshold sweep — is the EchoCache Space.

API

function	what it guarantees
normalize(text)	deterministic canonical form: NFKC, lowercase, whitespace and punctuation folding, greeting/sign-off/disclaimer removal (EN + JA)
exact_key(text)	sha256 of normalize(text) — the exact-match cache key
char_ngram_hash(text, n=3, dim=4096)	L2-normalized float32 vector; tokenizer-free, works on unspaced scripts
simhash(text, bits=64) / hamming(a, b)	fingerprint and bitwise distance for cheap candidate narrowing
should_cache(prompt, response)	(bool, reason_code) — refuses Luhn-valid card numbers, credential prefixes, JWTs, high-entropy secrets, PEM keys, excess contact PII
is_semantically_safe(query, candidate)	(bool, reasons) — the five MismatchGuard checks
estimate_tokens(text)	dependency-free token estimate (~4 ASCII chars or ~1 CJK char per token)
match(query, candidate, threshold, use_guard)	one complete reuse verdict

Behaviour is configurable through the same environment variables as the Space: DEFAULT_THRESHOLD, VECTOR_DIM, NGRAM_N, MAX_TEXT_CHARS, ENTROPY_THRESHOLD, MAX_EMAILS, MAX_PHONES, GUARD_DISABLE.

Self-test: python3 echocache_matcher.py (16 assertions, no network).

Limitations — read before adopting

• Surface similarity only. Paraphrases that share meaning but not characters are not matched, by design. Measured on this matcher: How do I reset my password? vs I forgot my login credentials, what now? = 0.12; What is the refund policy? vs Can I get my money back? = 0.00. If your traffic is full of synonym-only rewordings, this matcher alone will miss them — pair it with embeddings (the Space blends both when a token is configured) or accept the misses.
• The guard is conservative and will refuse valid reuse. Any difference in the numeric-token set vetoes a hit, so a 3 day trial and a three day trial do not match. That is the intended trade: a miss costs one API call, a wrong hit costs trust.
• Not a safety classifier. should_cache screens for secret-shaped strings. It is a last line of defence for a cache, not a DLP product, and it does not detect toxic or personal content beyond e-mail addresses and phone numbers.
• No cross-lingual matching. An English prompt never matches a Japanese one; character n-grams share nothing.
• Benchmark caveat. The benchmark was written by the same author alongside this matcher, and two matcher fixes were made in response to it. Treat the table above as a regression baseline, not an independent evaluation.

The three repositories

• Space — NagaYu/EchoCache: the running cache with dashboard, threshold sweep and JSON API.
• Model — NagaYu/echocache-matcher: this file.
• Dataset — NagaYu/echocache-guard-benchmark: the labelled pairs and the scoring script.

Apache-2.0.