Request access to RAVEN

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

RAVEN weights are licensed under CC BY-NC-ND 4.0; the accompanying code is licensed under MIT. Access to the model is limited to non-commercial academic research with proper attribution and is subject to the access terms below. Clinical, diagnostic, and therapeutic use is not permitted. Commercial use, sale, or other monetization of RAVEN, adapted models, models trained on its outputs, or datasets generated using it is prohibited without prior written approval from the authors. Commercial entities must contact the authors. Register through your own Hugging Face account and set its primary email to your institutional email. Provide your institution, principal investigator, and research purpose. By requesting access and downloading the model, you attest that this information is accurate and up-to-date and agree to the license, access terms, and safety guidance. Do not distribute, publish, or reproduce model weights for other users. Every user, including colleagues in your organization, must register individually and receive approval. Keep access tokens private. Do not attempt to re-identify training patients or reconstruct their records.

Log in or Sign Up to review the conditions and access this model content.

RAVEN: Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction

Preprint / GitHub

RAVEN is a transformer for longitudinal, structured electronic health records. This model uses a temporal decay of 0.5 and was trained for 600,000 microbatch iterations / 150,000 optimizer updates, with 42,351 token IDs and 33,193 supervised outputs. It predicts clinical features at visit separator positions using clinical tokens and elapsed days. Inputs are structured records, not free text.

Requesting access

Request access through your own Hugging Face account using your institutional email as the account's primary email. Provide your affiliation, principal investigator, and research purpose. Requests are reviewed individually for non-commercial academic research; each collaborator must submit their own request and receive approval. By requesting access and downloading the model, you attest that your information is accurate and up-to-date and accept the access terms. Read the License and Access terms and responsible use before requesting access. Commercial entities should contact the authors listed in Contact to discuss permission before using the model.

Load and run

Use Python 3.10 and install the validated dependencies from this folder:

python -m pip install -r requirements.txt
python example_inference.py --device cpu

The example uses fabricated feature IDs and dates. For CUDA, pass --device cuda after installing a compatible PyTorch/torchvision build.

import torch
from load_model import load_model
from example_inference import synthetic_inputs

model = load_model(device="cpu")
with torch.inference_mode():
    logits, _, _, _ = model(**synthetic_inputs())
print(logits.shape)  # [1, 4, 42351]

After downloading the entire Hugging Face repository, run the scripts from that downloaded folder. load_model() resolves metadata relative to its own file. The weights use the included native PyTorch architecture; a Transformers AutoModel, text-generation pipeline, and AutoTokenizer are not implemented.

Use all-zero patient_ids for a single patient. prepare_inputs() supplies these automatically. This argument represents the packed-window position, not a real patient identifier. Omitting it removes a positional embedding used in training and changes outputs. Use data_files/loss_mask_0_perc.npy to select supervised output IDs. Other output rows exist in the vocabulary but were not directly supervised as targets.

Model and training

Property Value
Architecture 8 transformer blocks, 8 attention heads, hidden width 1,024
Parameters 144,048,640, including fixed rotary frequencies
Context 512 tokens
Position encoding Day-based rotary embeddings; XPos disabled
Embedding/output weights Tied
Vocabulary 42,345 retained columns plus 6 sequence controls
Supervised outputs 33,193, occurring in more than 100 distinct patients
Loss Sigmoid focal loss, gamma 2.0, alpha 0.995
Temporal decay 0.5; repeated positive targets have weights 1, 0.5, 0.25, 0.125, then 0.1
Training From scratch on NYU structured EHR data; multi-window packing
Optimizer AdamW; learning rate 0.00022; weight decay 0.01
Training compute Two A100 GPUs; bfloat16 autocast
Released weights Original FP32 tensors; no quantization

The decay is applied to repeated positive occurrences of each target within a sampled window, with a minimum weight of 0.1. The relative-day column and six sequence controls are excluded from the loss. Eligible outputs comprise 23,937 diagnoses, 6,804 medications, 2,408 lab bins, and 44 demographic targets.

config.json records the architecture and loss settings, temporal decay, training iteration counts, and tensor aliases needed to load this release.

Using your own data

Map your clinical records directly to the frozen model vocabulary. Each JSON list index corresponds to model token ID index + 6. load_vocabulary() adds the six sequence controls so its returned list can be indexed directly by token ID. The label order is fixed by the weights and must not be changed.

Group the mapped clinical IDs into visits and supply their elapsed days. prepare_inputs() adds separators, day embeddings, attention masks, and the single-patient position indices expected by the model:

import torch
from load_model import load_model, load_vocabulary
from prepare_inputs import prepare_inputs

labels = load_vocabulary()
token_to_id = {label: token_id for token_id, label in enumerate(labels)}
# Example only: replace this with features mapped from your own records.
visits = [[token_to_id["Diagnosis:Essential (primary) hypertension I10"]]]
inputs = prepare_inputs(visits, visit_days=[0], prediction_day=365)
model = load_model()
with torch.inference_mode():
    logits, _, _, _ = model(**inputs)
forecast_logits = logits[0, -1]  # Last SEP predicts at prediction_day.

Your mapping must account for clinical codes, medication identities, lab units and normalization, age bins, and missing observations. See PREPROCESSING.md for the encoding rules and input contract. The helper operates on already-mapped IDs; it does not infer equivalence between your local codes and the vocabulary labels.

The only files in data_files/ are the vocabulary and the supervised-output mask. The mask distinguishes the 33,193 trained prediction targets from other vocabulary slots; clinical features outside that target set may still be used as inputs.

Evaluation

The table below summarizes performance on six EHRSHOT tasks. EHRSHOT condition headers and their model token IDs define the disease scores.

Evaluation used FP32, context 512, batch size 16, all-zero patient-position indices, direct prediction at 365 days, and sums of disease logits. The six tasks cover 11,387 label instances; counts are task-specific and should not be treated as unique patients across tasks.

Condition AUROC AUPRC
Heart attack 0.6862 0.1370
Pancreatic cancer 0.7838 0.1857
Hypertension 0.7022 0.2523
Hyperlipidemia 0.6262 0.2045
Lupus 0.6707 0.0412
Celiac disease 0.5720 0.0116

Intended use

The intended use is research on longitudinal clinical representation and prediction. Examples include studying clinical representations, evaluating future-feature prediction, and comparing research methods on appropriately governed datasets.

Limitations & Safety

RAVEN is a research model. This release does not support clinical, diagnostic, or therapeutic use, and its evaluation does not establish regulatory clearance or clinical effectiveness. Do not use its outputs to make patient-care decisions.

The NYU training cohort and its documentation practices can introduce demographic and clinical biases. Performance may change across institutions, patient groups, coding systems, lab units, observation schedules, and missing-data patterns. The six reported EHRSHOT tasks do not establish performance for every population, condition, or use case. Evaluate subgroup performance and the intended setting independently before drawing conclusions.

Outputs are uncalibrated scores and can be wrong. Rare-outcome estimates have few positive patients; assess uncertainty in the setting where you evaluate the model. Scores do not establish that a patient has, or will develop, a condition. If scores are incorporated into an LLM or another system, independently check the resulting claims; plausible prose does not validate a prediction.

The release contains no patient-level records. Model weights are not a guarantee of privacy: do not attempt to identify training patients or reconstruct their records. Use only data you are authorized to process, and do not send identifiable patient information through public issues, discussion threads, or support email.

Included files

model.safetensors contains only model tensors. The output projection shares the embedding weights; config.json records the alias and load_model.py restores it before strict state loading. The optimizer and training checkpoint pickle are omitted. This follows the safetensors treatment of shared tensors.

model.py, focal.py, and rotary_embedding_torch.py implement the trained architecture. load_model.py loads the weights and vocabulary; prepare_inputs.py builds inputs from mapped visits using special_tokens_map.json for sequence controls. example_inference.py provides a runnable synthetic example, and PREPROCESSING.md describes how to encode your own data. requirements.txt records the validated dependencies. data_files/ contains the vocabulary and output mask, and evaluation/ contains EHRSHOT condition headers and their model token IDs.

The source project acknowledges nanoGPT; the bundled rotary implementation is derived from rotary-embedding-torch. Their notices are retained in THIRD_PARTY_NOTICES.md.

Contact

For research questions, access requests, or licensing inquiries, contact:

License

The code's MIT license applies to the accompanying code; the weights have their own license. The full license texts govern copyright permissions. Access to the hosted model also requires acceptance of the terms below. Contact the authors for commercial-use requests or other licensing inquiries.

Access terms and responsible use

By requesting access and downloading the model from this repository, you agree to the following access terms:

  1. Permitted use and attribution: Use RAVEN only for non-commercial academic research. Please provide proper attribution to the original authors, including, but not limited to, citing the paper below, identifying the model version in research reports, and retaining the applicable license and attribution notices. Clinical, diagnostic, and therapeutic use is not permitted.
  2. Commercial prohibition: Commercial use, sale, or other monetization of RAVEN requires prior written approval from the authors. This restriction also applies to adapted models, models trained on RAVEN outputs, and datasets generated using RAVEN. Commercial entities must contact the authors before use.
  3. Access, registration, and accuracy: Register on Hugging Face using your own account, with your institutional email as its primary email, and accept the license and these terms before downloading. Provide your institution, principal investigator, and research purpose. You attest that your submitted information is accurate and up-to-date and will keep it current.
  4. Non-distribution: Do not distribute, publish, or reproduce copies of the model weights for other users. This includes sharing weights with colleagues within your organization. Direct prospective users to request access here.
  5. Individual access: Approval is personal. Every user must register independently, accept these terms, and receive approval. Keep access tokens private; do not share accounts, credentials, or access approval.
  6. Data privacy: Do not attempt to re-identify training participants, reconstruct or extract their records, or use the model to disclose private patient information. Process only data you are authorized to use.
  7. Safety and warranties: Follow the safety guidance above. The model is supplied without warranties of accuracy or fitness for a particular purpose, subject to the full license terms.

Citation

Please cite the following paper when using RAVEN in your research:

@article{gao2026raven,
  title={Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction},
  author={Gao, Xiang and Rajamohan, Haresh Rengaraj and Zhu, Weicheng and Huang, Shih-Lun and Chen, Long and Schulman, Gabe and Jin, Huizhen and Li, Shengduo and Wang, Yixuan and Yang, Huidi and others},
  journal={arXiv preprint arXiv:2603.24562},
  year={2026}
}
Downloads last month
-
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Paper for NYUMedML/RAVEN