abr00kx/forensic-suite

Forensic Suite

A stylometric, temporal and network-analysis toolkit for public forum archives. It harvests threads, extracts writing-style features, profiles posting cadence, clusters posters by multi-factor similarity, and serves the results through a CLI and a web dashboard.

Read this first. See Scope, limits and ethics. Every score this tool produces is a statistical candidate, not an identification.

Status

Go owns everything except the data-science layer. That split is now in place end to end:

Area	Where it lives	Language
Harvesting (4chan / DesuArchive)	internal/harvest	Go
Post storage	internal/store	Go
Dashboard HTTP server and JSON API	internal/api, cmd/forensic-suite serve	Go
CLI (harvest, identify, serve)	cmd/forensic-suite	Go
Stylometry, temporal, clustering, drift	src/forensic_suite/pyds/analytics/	Python
Bridge the Go server calls	src/forensic_suite/pyds/analytics/bridge.py	Python
Pandas/HDBSCAN profiling	src/forensic_suite/pyds/	Python
Dashboard frontend	web/static/	HTML/CSS/JS

The Python web server has been retired: src/forensic_suite/pyds/web/ is gone and python -m forensic_suite.pyds web no longer exists, because a second HTTP implementation is exactly the duplication that made "which file is live?" unanswerable. The dashboard is served by the Go binary, which delegates the statistical endpoints to Python over a JSON bridge.

The Python CLI now carries only the analysis subcommands (scan, report, temporal, sockpuppets, drift, enrich) — that is the data science, and it stays Python. The legacy per-board harvesting and enrichment scripts that used to sit under analytics/ have been retired: forensic-suite harvest replaces them, and python -m forensic_suite.pyds harvest no longer exists.

Scope, limits and ethics

This tooling analyses posts written under pseudonyms on an anonymity-focused imageboard. That framing has consequences that are easy to lose sight of once there are numbers on a screen.

• Similarity is not identification. A cosine score means two writing samples resemble each other more than a baseline comparison. It does not establish common authorship and it does not establish any real-world identity.
• Poster IDs are weak identifiers. They are scoped per thread, expire, and are reused. Treating an ID as a stable person is an error.
• The corpus is incomplete. Threads expire and harvesting is partial, so absence of a signal is not evidence of absence.
• Scores are not comparable across runs. They depend on the corpus present when they were computed.
• Small samples produce confident nonsense. Short posts and low post counts yield unstable feature vectors.

The intent of this project is archive analysis and stylometric methodology, not deanonymisation of individuals. If you intend to act on a result in a way that affects a real person, stop — the output does not support it.

The same wording is attached programmatically to API responses as provenance.caveats, and rendered permanently in the dashboard UI, so a score cannot be lifted out of context without its caveat. Both halves emit it: internal/provenance/provenance.go for Go-served responses and src/forensic_suite/pyds/analytics/provenance.py for delegated ones.

Provenance

Every analytical response carries a provenance block recording how it was produced:

{
  "tool": "forensic-suite",
  "version": "1.0.0",
  "generated_at_iso": "2026-09-13T12:04:06Z",
  "inputs": {
    "posts_db": { "path": "...", "exists": true, "size_bytes": 3121152, "modified": 1789294612 }
  },
  "parameters": { "confidence": null, "limit": 50 },
  "caveats": { "candidate_not_identification": "...", "corpus_limits": "..." }
}

This ties a result to the exact corpus it was computed from and to every threshold that shaped it.

Installation

Go components need Go 1.27+:

go build -o forensic-suite ./cmd/forensic-suite

The Python data-science layer is only needed for the statistical endpoints. Its dependencies are pinned in requirements.txt (numpy, pandas, scikit-learn, hdbscan); nothing else in the repo needs a Python install.

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

The Go server discovers an interpreter in venv/ or .venv/ first, then falls back to python3/python on PATH. Pass --python to override. If no interpreter is available the server still starts; the delegated endpoints answer 503 and explain why.

Data

The tooling reads a SQLite database of harvested posts (default data/databases/bant_analysis.db). Resolution order, implemented in forensic_suite.pyds.analytics.stylometric_classifier.resolve_db_path:

1. BANT_DB_PATH environment variable
2. data/databases/bant_analysis.db at the repository root
3. bant_analysis.db in the working directory

Related environment variables:

Variable	Purpose
BANT_DB_PATH	Posts database
DETECTIONS_DB_PATH	Detections database (default data/databases/detections.db)
DETECTOR_LOG_PATH	Detector log file (default data/logs/detector.log)
THREAD_DOWNLOADS_DIR	Directory of raw thread_*.json dumps (default data/thread_downloads/)

Go resolves the posts database with the same order (environment override, then the repository root) in internal/config, and the detections database from DETECTIONS_DB_PATH or data/databases/detections.db at the repository root. serve passes both paths to the Python child, so the two halves always read one corpus.

Databases, scraped threads and generated artefacts are gitignored: they contain verbatim posts by real users and should not be committed.

CLI

Go (current)

# Build once
go build -o forensic-suite ./cmd/forensic-suite

# Search board catalogs for threads matching patterns and store them
./forensic-suite harvest --boards bant,pol --patterns heh,gosh --max-threads 30

# Respect the API: it allows about one request per second. The default matches
# that; raising it will get you throttled.
./forensic-suite harvest --requests-per-second 1

# Pivot from a poster ID to IDs sharing its images on DesuArchive
./forensic-suite identify pMhJ34f4 --out resolved_identities.json

# Media Forensics: Binary JPEG DQT table extraction & device profiling
./forensic-suite media -filename "Screenshot_2026-09-11_at_6.33.30_PM" -ts 1789166452
./forensic-suite media -stats -db master_intelligence.db
./forensic-suite media -backfill -db master_intelligence.db

# Open-World Multi-Perspective Verification (SOTA N-grams, Syntax, Unmasking, Bayes Factor)
./forensic-suite verify -poster1 USER_averi.foo_Lachlan -poster2 "MATRIX_@midov:midov.pl" -db master_intelligence.db
./forensic-suite verify -text1 "Sample writing from author A..." -text2 "Sample writing from author B..."

# Historical Archive Search (FoolFuuka / Asagi API)
./forensic-suite archive -tripcode "!zcleypcJXz2" -boards g,bant -db master_intelligence.db

# Invariant Image Matching (64-bit DCT pHash)
./forensic-suite phash -hash1 "a1b2c3d4e5f60718" -hash2 "a1b2c3d4e5f60719"

./forensic-suite help

See SCIENTIFIC_METHODOLOGY.md for full mathematical definitions, peer-reviewed literature citations (NAACL, EACL, JASIST, PACLING, arXiv), and BibTeX entries.

Harvesting is idempotent: post_no is uniquely indexed, so re-running a harvest inserts nothing new. It never drops or rewrites the posts table.

Python (transitional)

python -m forensic_suite.pyds --help

# Stylometric scan of text or a poster
python -m forensic_suite.pyds scan --text "Well heh gosh this is a test post ^^"
python -m forensic_suite.pyds scan --poster pMhJ34f4

# Interactive HTML dashboard
python -m forensic_suite.pyds report -o dashboard.html

# Temporal activity, sessions and cadence
python -m forensic_suite.pyds temporal --poster pMhJ34f4 --sessions

# Sockpuppet candidates and clusters
python -m forensic_suite.pyds sockpuppets --min-posts 3 --min-composite 0.5

# Behavioural drift over time windows
python -m forensic_suite.pyds drift --poster pMhJ34f4 --window-days 7 --threshold 0.2

# Thread poster-ID enrichment
python -m forensic_suite.pyds enrich --limit 50

The dashboard server is no longer here: run ./forensic-suite serve, and see Web dashboard.

Web dashboard

./forensic-suite serve --addr 127.0.0.1:8080

It serves web/static/ and exposes the JSON API the frontend calls. Every endpoint that computes a statistic is delegated to the Python data-science layer by spawning python -m forensic_suite.pyds.analytics.bridge <command> and passing a single JSON object each way.

Flags: --addr, --db, --detections-db, --watchlist, --static, --repo-root, --python. Run ./forensic-suite serve -h for the defaults.

It binds 127.0.0.1 by default and has no authentication. It serves a corpus of real users' posts; do not expose it to a network.

Which half does what:

Endpoint	Served by
/, /static/*	Go (static files)
/api/overview, /api/posters, /api/threads*, /api/watchlist*, /api/detections	Go (SQLite)
/api/harvester/catalog, /api/harvester/run	Go (board API)
/api/posters/{id}, /api/posters/compare/{a}/{b}, /api/network	Python (delegated)
/api/sockpuppets, /api/scan, /api/export/report	Python (delegated)

Go relays delegated payloads verbatim, so a response shape is defined in one place and the two halves cannot drift apart.

Project layout

.
├── go.mod                      # single Go module: github.com/ashley/forensic-suite
├── cmd/
│   └── forensic-suite/         # unified CLI (harvest, identify, serve)
├── internal/
│   ├── api/                    # dashboard HTTP server, watchlist, delegation
│   ├── config/                 # database path resolution
│   ├── ds/                     # JSON bridge client for the Python layer
│   ├── harvest/                # 4chan + DesuArchive clients, rate limiting, backoff
│   ├── model/                  # shared post/thread types
│   ├── provenance/             # provenance records and the caveat wording
│   ├── store/                  # SQLite schema, idempotent inserts, read queries
│   └── text/                   # HTML cleaning (mirrors the Python implementation)
├── src/forensic_suite/pyds/          # Python data-science layer (see Status)
│   ├── analytics/              # Stylometry, temporal, sockpuppet, drift, provenance
│   │   ├── bridge.py           # entry point the Go server calls
│   │   └── analysis_api.py     # response shapes the dashboard renders
│   ├── domain/                 # Domain-layer scaffolding (largely not wired up yet)
│   └── cli.py                  # Python analysis CLI
├── src/forensic_suite/pyds/    # Pandas/HDBSCAN profiling
├── web/static/                 # Dashboard frontend (HTML/CSS/JS)
├── tests/                      # Python suite (hermetic; no network, no real DB)
└── requirements.txt, pyproject.toml

The Go code carries its own tests alongside each package (*_test.go), and internal/api/server_test.go covers the HTTP contract the frontend depends on.

Development

gofmt -l . && go vet ./... && go test ./...        # Go
gofmt -w ./internal ./cmd                         # Go formatting
pytest                                            # Python suite
black src tests                                   # Python formatting
flake8 src tests                                  # linting
mypy src/forensic_suite/pyds/analytics

Both suites are hermetic: they build their own temporary fixture databases (BANT_DB_PATH / DETECTIONS_DB_PATH) and perform no network I/O. Do not point tests at your real corpus.

tests/test_bridge.py runs the bridge as a subprocess with a pinned database, because that is how the Go server calls it, and it cross-checks that the commands the Go server sends match the handlers this package defines. If you add a delegated endpoint on one side, add it on the other or that test fails.

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). See the LICENSE file for the complete license terms and conditions.

Under the AGPL-3.0 copyleft terms, any modified versions deployed over a computer network or integrated into cloud-hosted APIs must make their complete corresponding source code available to all interacting users.