rasynai/rasyn-iris

How it works · Quickstart · Results · Configuration · Notes from development

---

IRIS takes a natural language literature query like

"Find papers that use diffusion models for protein structure generation, evaluate on CASP targets, and were published after 2022."

and returns a ranked list of papers where every stated constraint has been checked against the paper's actual text, with verbatim evidence attached to each result.

It runs as "Rasyn IRIS" on Ai2's AstaBench leaderboard, scored under the official harness.

Results

Official AstaBench PaperFindingBench (validation split, astabench 0.5.4, live GPT-4o grading):

System	Adjusted F1	Cost / query	Stack
Asta Paper Finder (Ai2, SOTA)	0.433	~$0.35	closed pipeline
IRIS (this repo)	0.386	~$0.12	gpt-4o-mini + gpt-5-mini

All numbers come from official harness runs, not offline estimates.

How it works

flowchart LR
    Q([Query]) --> R{Router}
    R -->|semantic| F[Multi-channel retrieval<br/>snippet · keyword · citations]
    F --> J[Per-criterion pool judge<br/>gpt-4o-mini, depth 400<br/>position-decay weighting]
    J --> T[Listwise tournament<br/>gpt-5-mini, pooled Borda<br/>bounded demotion]
    R -->|metadata| M[Deterministic plan executor<br/>+ 923M-edge citation graph]
    R -->|specific| S[Targeted lookup]
    T --> O([Ranked papers +<br/>verbatim evidence])
    M --> O
    S --> O

One cheap LLM call routes each query to the channel built for it: semantic (fuzzy topical), metadata (venue, year, author, or citation constraints), or specific (a known paper). Deterministic channels that come back empty fall through to semantic, since an empty answer scores zero and the judged channel can only help.

Semantic search

The semantic channel fans out across snippet search, keyword search, and citation expansion, then fuses everything with reciprocal rank fusion. The query is decomposed into explicit relevance criteria, and a gpt-4o-mini pool judge scores the top 400 candidates against each criterion separately, with position-decay weighting (later auto-derived criteria are noisier, so they count less). Pointwise scores strand real matches in the ambiguous middle, so a gpt-5-mini listwise tournament reorders that contested band with sliding windows, pooled Borda aggregation across passes, and a bounded demotion cap so one bad window cannot destroy a good paper's rank.

Metadata search

"Papers at CHI after 2020 citing NeurIPS papers" is a database query, not a similarity search. The metadata channel compiles the query into a typed plan (venue set, year filter, citation set) and executes the intersection deterministically, with venue canonicalization and acronym expansion on both sides. When available, the citation constraint runs against a local 923 million edge Semantic Scholar citation graph instead of rate-limited APIs.

Specific lookup

When the query names one paper ("the AlphaFold 2 Nature paper"), IRIS extracts every clue it can (title fragments, authors, year, topic, artifacts), scores candidates against all of them, walks references when clues conflict, and returns a single verified paper with verbatim evidence.

Quickstart

git clone https://github.com/rasynai/rasyn-iris.git
cd rasyn-iris
pip install astabench==0.5.4

Create iris_asta/.env (see .env.example):

OPENAI_API_KEY=sk-...
ASTA_TOOL_KEY=...        # free at https://api.semanticscholar.org

Run the full benchmark under the official harness:

inspect eval astabench/paper_finder_validation \
  --solver pfbmax/inspect_entry.py@pfbmax_solver \
  --model openai/gpt-4o-mini

Or call IRIS from your own code:

import sys; sys.path += ["pfbmax", "iris_asta"]
from iris_asta.asta_client import AstaClient
from iris_asta.config import load_config
from llm import LLM
import router

client = AstaClient(load_config())
results = router.solve(
    "diffusion models for protein structure generation evaluated on CASP, after 2022",
    client, LLM(), inserted_before=None,
)
for paper_id, evidence in results:
    print(paper_id, evidence[:100])

router.solve never raises. It routes, retrieves, judges, and returns [(corpus_id, verbatim_evidence), ...] best first.

Configuration

Everything is tunable by environment variable. The config that produced the leaderboard score:

Variable	Value	What it does
PFBMAX_CJ_POOL	1	enable the per-criterion pool judge
PFBMAX_CJ_POOL_DEPTH	400	candidates judged per query
PFBMAX_CJ_POSDECAY	0.6	criterion position-decay weight
PFBMAX_TOURN	1	enable the listwise tournament
PFBMAX_TOURN_MODEL	gpt-5-mini	tournament ranking model
PFBMAX_TOURN_PRIOR	0.4	blend weight of the pointwise prior
PFBMAX_TOURN_DEMOTE_CAP	8	max ranks a paper can fall per tournament
PFBMAX_CITEGRAPH / PFBMAX_PMETA	paths	optional local citation graph + metadata SQLite

Repository layout

Path	What lives there
pfbmax/router.py	query classification and channel dispatch, start here
pfbmax/criterion_judge.py	the per-criterion pool judge
pfbmax/tournament.py	listwise tournament reranker
pfbmax/metadata_solver.py	deterministic metadata plans and citation graph execution
pfbmax/inspect_entry.py	official harness entry point
iris_asta/	corpus client, config, rate limiting, snapshot date enforcement
analysis/	offline scorer replicating the benchmark metric

Notes from development

Every component here earned its place through a controlled experiment, and everything that lost one was deleted. The campaign ran 23 documented experiments and six official harness runs. Things that did not work, so you do not have to retry them:

• Trained cross-encoder rerankers (4 variants): never beat the LLM judge
• Bradley-Terry and PageRank aggregation: pooled Borda won
• Hierarchical tournaments: flat sliding windows won
• Permissive judge prompts, evidence enrichment, prior blends at admission: all net negative

Things that paid: position-decay criterion weighting, pooled cross-pass Borda, demotion caps, the local citation graph.

The tournament's demotion cap is a good example of the approach. Pilot runs showed one rescue worth +0.094 but two catastrophic demotions, so ascent is unlimited and descent is capped at 8 ranks. That single asymmetry took the feature from net zero to net positive.

Contributing

Issues and PRs welcome. One rule: no change lands without a measured comparison. Run the offline scorer in analysis/ against the validation references and post the delta.

License

MIT © 2026 Rasyn AI

---

_{Built by Rasyn AI · Benchmarked on AstaBench by Ai2 · Mirrored on Hugging Face}