BERURIA

An English-native encoder that understands Torah vocabulary.

BERURIA stands for BERT English Representations for Usage in Rabbinic Information and Analysis. It is ModernBERT-base with its masked-language-model pretraining continued on Sefaria's English corpus. The goal is a general-purpose English encoder that knows the vocabulary of Jewish texts: names of sages and books, Hebrew and Aramaic terms transliterated in English prose, and the citation and commentary conventions of the Jewish library.

General English encoders see this material rarely, so they split terms like Mishneh Torah, Tosafot or Ketubot into meaningless fragments and know little about how they are used. This model is meant as a better starting point for fine-tuning on English-language Jewish text, for example NER (people, citations), classification, retrieval and linking.

Details

Base model	answerdotai/ModernBERT-base (22 layers, hidden size 768, 8192 max positions)
Objective	Masked language modeling, 30% mask probability
Training data	Sefaria/BERURIA-training-data: English text from Sefaria's library, source sheets and related web pages (266,654 documents, ~209M words), packed into 1024-token windows
Training	10 epochs (~2.9B tokens), peak learning rate 6e-4 (linear decay, 5% warmup), global batch of 512 sequences, 1x A100 40GB, FlashAttention 2
Final eval loss (MLM)	1.176
Tokenizer	ModernBERT's tokenizer extended with 6,000 Hebrew BPE tokens (vowel marks stripped by the normalizer). English tokenization is unchanged from the base model, and the new embeddings were initialized from the mean of their old subtokens. The embedding matrix has 56,368 rows.

The Hebrew tokens let the model read Hebrew quoted inline in English text without shattering it, but the model is trained on English text, so it is not a Hebrew language model.

Usage

from transformers import pipeline

fill = pipeline("fill-mask", model="Sefaria/BERURIA")
fill("Rashi was a medieval [MASK] who wrote a commentary on the Torah.")
# commentator, scholar, sage, ...

For fine-tuning, load it like any ModernBERT checkpoint:

from transformers import AutoModelForTokenClassification, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Sefaria/BERURIA")
model = AutoModelForTokenClassification.from_pretrained("Sefaria/BERURIA", num_labels=5)

Requires transformers>=4.48.

Limitations

• This is a pretrained encoder, not a task model. Fine-tune it for your task.
• Training text is English. Performance on purely Hebrew or Aramaic input is not what this model is for.
• The model has absorbed the viewpoints and register of its training corpus, which is largely traditional Jewish text and commentary.