Takzen/jay-4b

Jay 4B (sójka)

Jay 4B reads one npm package version, its package.json, file list and the code most likely to run, and answers two questions with a probability for every option: is this version malicious? and, if it is, was the package built to attack, or is it a hijacked library?

It is a LoRA adapter on Qwen/Qwen3.5-4B that runs in 4-bit on one consumer GPU. Each answer comes from the option-letter logits of one forward pass; no text is generated. Sibling of Shrike 4B, Bullfinch 4B and Serin 4B.

Why. Malicious npm versions keep coming: DataDog's open dataset holds 25,460 versions built to attack and 1,448 hijacked releases published from 2024 to October 2026. Blocking install scripts (pnpm 10 does it by default) stops the attacks that run from a preinstall or postinstall hook, but in 2026 35% of the attack versions in that dataset run when the package is imported instead, with no hook at all. Jay reads the code, so it does not care how the code gets to run. It is a triage aid: it ranks what a person should look at first.

$ python jay_infer.py esbuild@0.28.2
esbuild@0.28.2  (7 files, install hooks: postinstall)
verdict: benign  p(malicious) = 0.017
code shown to the model: install.js, lib/main.js, bin/esbuild

$ python jay_infer.py chai-as-repaired-5.32.9.tgz --registry-doc chai-as-repaired-5.32.9.json
chai-as-repaired@5.32.9  (42 files, install hooks: none)
verdict: malicious  p(malicious) = 0.994
origin:  attack  p(hijacked) = 0.003
code shown to the model: index.js, lib/meta.js, lib/deprecations.js, lib/const.js, lib/time.js, file.js, lib/constants.js, lib/symbols.js, lib/transport-stream.js

esbuild fetches its platform binary from a postinstall script, the kind of benign hook an earlier pilot flagged (0.85 on esbuild's latest release). chai-as-repaired 5.32.9 is a 2026 test sample from the DataDog dataset, built to attack and without an install hook: its code runs when the package is imported.

Results

Test panel: 1,992 npm packages first seen in 2026 and never trained on (453 malicious versions, 1,622 benign), the same rows for every method. Flags at p(malicious) ≥ 0.5; GuardDog at its own risk labels.

Method	AUROC	Caught: built to attack	Caught: hijacked	False alarms: random npm	False alarms: with an install script
Rule: has an install script	0.736	62%	57%	2.8%	99.5%
GuardDog 3.2.0, any risk	0.820	84%	80%	23%	89%
GuardDog 3.2.0, high risk (≥ 7.6)	0.820	40%	46%	3.5%	16%
Qwen3.5-4B, untuned	0.850	71%	6.7%	0.1%	4.9%
Jay 4B	0.927	81%	40%	1.3%	3.3%

• AUROC +0.107 over GuardDog (95% interval 0.084 to 0.130) and +0.077 over the untuned base (0.056 to 0.097), paired bootstrap by package on the same rows.
• Origin. Of the malicious versions, 94.3% are correctly told built to attack or hijacked (macro-F1 0.916; the untuned base 41%).
• Popular packages. No false alarm on 197 versions of popular npm packages published in 2026 (names never seen in training; the install-script rule flags 12% of them), and none on 86 well-known packages at their latest release (esbuild, electron, bun and others).
• Calibration. Temperature 1.231, fitted on a calibration split of 974 answers (ECE 1.3% to 0.8%). At a calibrated p ≥ 0.95 Jay still catches 70% of the malicious versions, with 4 false alarms in 1,622 benign versions (0.25%).
• Where it loses: hijacked libraries. Jay catches 40% of them; GuardDog at any risk catches 80% and the install-script rule 57%, each with far more false alarms. A hijacked release is a real, often large library with injected code, and Jay sees about 6,000 characters of code. The first pilot caught 52% of hijacks but flagged 14% of benign packages with an install script, esbuild, electron and bun among them; this release gives up hijack recall for that.

How to use

jay_infer.py in this repo is self-contained (torch, transformers, peft, bitsandbytes, accelerate; a CUDA GPU):

python jay_infer.py left-pad@1.3.0            # fetched from registry.npmjs.org, integrity checked
python jay_infer.py ./some-package-1.0.0.tgz  # a local tarball
python jay_infer.py express@5.1.0 --json      # full distributions, files shown, install hooks

The tarball is read in memory: nothing is extracted, installed or run. The prompt is a chat message with the package as the state (package.json, the file list with sizes, then the install-hook targets, the entry points and the other code files by size, about 6,000 characters of code in all), the question and the lettered options with their definitions; the assistant turn is prefilled with {"answer": " and the decision is a softmax over the logits of the option letters, divided by the fitted temperature (in CALIBRATION.json). The origin question is asked only when p(malicious) ≥ 0.5.

• Speed. One forward pass per question, up to ~3,600 tokens; about one question per second on a consumer GPU in WSL.
• Where it fits. Score only what is new: versions that appear in a lockfile diff, keyed by name, version and integrity hash. A version whose hash differs from the public registry's deserves an alert of its own before any model reads it.

Training

• QLoRA: base frozen in 4-bit NF4, LoRA r 32, alpha 64, dropout 0.05 on the attention, Gated DeltaNet and MLP projections.
• 7,000 draws planned (65% the verdict question, 35% the origin question with attack and hijacked balanced), batch 16, learning rate 1e-4 with cosine decay, prompts up to 3,584 tokens. Versions of popular npm packages (1,260 from 2023 to 2025, names kept out of the test set) are drawn twice as often as other benign versions.
• The run was stopped at step 260 of 438; the adapter kept is step 250 (4,000 examples seen), the lowest NLL on the development panel (0.067). About 5 hours on one desktop GPU, evaluations included.
• Every example is rendered with a random option order; each question has three phrasings.
• Two runs, no paid compute. The first (6,000 draws, no popular packages) reached AUROC 0.946 but flagged 14% of benign packages with an install script; the second, this release, 3.3%.

Data

• Malicious: npm samples from DataDog's malicious-software-packages-dataset (Apache 2.0): malicious_intent = built to attack, compromised_lib = hijacked library. Read in memory from the password-protected zips; never extracted or run.
• Benign, three sets: 15,000 random npm versions published 2024 to October 2026, drawn from the registry's replication feed (names that DataDog or OSV's malicious-packages list mark as malicious excluded); 1,797 random versions that ship an install script, so that "has a hook" cannot stand in for "malicious"; and 931 clean versions of the hijacked libraries, the last release before each attack.
• Versions with no code that could run (an empty package, or code nothing loads) are dropped on both sides.
• Split by date: packages first seen in 2026 are the test set; earlier ones are training, with two 5% slices (by package name) for development and calibration. A package never crosses splits; a hijacked library's clean version goes with its attack. Campaigns that publish the same code under many names are capped (one version per package and per code fingerprint in the test set).
• The training data is not released: it contains live malware.
• No outputs of commercial models or decision APIs were used.

Limitations

• npm only. PyPI is not covered yet.
• Hijacked libraries: 40% caught on the test panel (see Results).
• It judges what it is shown: about 6,000 characters of code, picked by install hooks, entry points and size. Code past that budget, inside binaries, or downloaded at run time is not seen.
• Label bias. DataDog finds most of its samples with GuardDog, so the malicious side leans towards what GuardDog detects; malware that GuardDog misses is rare in training and test, and recall in the wild may be lower. Benign versions are random registry versions filtered against DataDog's and OSV's lists; a few may be malicious and unknown.
• One time slice. The test set is January to October 2026; new techniques will drift.
• Beacons count as malicious: bug bounty and dependency-confusion proofs of concept that report the host to a callback domain are labelled malicious, as in the dataset.
• Adversarial text aimed at the model inside a package (comments claiming it is safe) was not tested.
• It is a triage aid: p(malicious) < 0.5 does not mean safe, and every flag needs a person.

Licence and attribution

• The adapter and jay_infer.py are released under the Apache License 2.0. The base model is Qwen3.5-4B (Apache 2.0).
• Malicious samples come from DataDog's malicious-software-packages-dataset (Apache 2.0). The GuardDog baseline is DataDog's GuardDog 3.2.0 (Apache 2.0).
• npm is a trademark of npm, Inc. Jay is not affiliated with or endorsed by npm, GitHub or DataDog.