Nomic Embed: Training a Reproducible Long Context Text Embedder
Authors: Zach Nussbaum
John Xavier Morris
Andriy Mulyar
Brandon Duderstadt
Venue: Accepted by TMLR ()

Abstract: This technical report describes the training of nomic-embed-text-v1, the first fully reproducible, open-source, open-weights, open-data, 8192 context length English text embedding model that outperforms both OpenAI Ada-002 and OpenAI text-embedding-3-small on the short-context MTEB benchmark and the long context LoCo benchmark.
We release the training code and model weights under an Apache 2.0 license.
In contrast with other open-source models, we release the full curated training data and code that allows for full replication of nomic-embed-text-v1. You can find code and data to replicate the model at \href{https://github.com/nomic-ai/contrastors}{https://github.com/nomic-ai/contrastors}

Published in Transactions on Machine Learning Research (02/2025)
Nomic Embed: Training Reproducible Long Context Text Embedder
Zach Nussbaum Nomic AT
zoch @nomi.at
John X. Morris
jack@nomic..e ai, jema&cornellledi
Nomic AI, Cornell Universiby
Brandon Duderstadt
brandon@nomic.at
Nomic AT
Andriy Mulyar
andriy@nomic. et
Nomic AI
Reviewed on OpenReview: https: // openreview. net/ forum? id- PPmzyQS4Q5
Abstract
This technical report describes the training of nomic-embed-text-v1. the first fully repro- ducible, open-source, open-weights, open-data, 8192 context length English text embed- ding model that outperforms both OpenAI Ada-002 and OpenAI text-embedding---smae on the short-context MTEB benchmark and the long context LoCo benchmark. We release the training code and model weights under an Apache 2.0 license. In contrast with other open-source models, we release the full curated training data and code that allows for full replication of nomic-embed-text-vl.. You can find code and data replicate the model at http:://ggluubb.comm/mmmic////ss
1
Introduction
Text embeddings are an integral component of modern NLP applications powering aetrivval-ungmenter generation (RAG) for LLMs and semantic search (Lewis et al. 2021a. Izacard et al. 2022bb Ram et al. 2023 These embeddings encode semantic information about sentences as low-dimensional vectors that are used reamstream applications, such as clustering for data visualization, classification, and information retrieval.
The majority of the top open-source models the MTEB benchmark MMuennighoff et al. 2023) are limited to context lengths of 512, such as E5 (Wang et al. 2022), GTE (Li et al. 2023), and BGE (iia et al. 2023
This short context length reduces model utility in domains where the overall document semantics are not localized to sentences or paragraphs. Most top embedding models with context length longer than 2048
are closed-source, such as Sorage-lite-01-insttie et al. 2022).
VVyagee 2023 and httt--embbddinn--ddd------- Neelakantan
As of October 2024, the lop-performing open-source long context embedding models are jina-embedding: v2-base-en (Günther et al.) 2024) and E5-Mistral--7-insttu (Wang et al. 2023b). Unfortunately, jina- enbedding-v2--aar does not surpass OpenAT's poot-embedding-addaooo
(Neelakantan et al. 2022) (sec Table
Further, E5-Mistral (Wang et al.) 2023b) is not feasible to use in many engineering applications due to the large inference requirements a 7 billion parameter transformer, and does not. perform well beyond 4096 tokens.
In this paper, we present an end-to-end training pipeline for state of the long context text embedding model at only 137 million parameters. nomic-embed-textte outperforms OpenAl text-embedding-ade and

---

Published in Transactions on Machine Learning Research (02/2025)
Nomic Embed Jina Base V2
62.39
text-ombedding---mal
60.39
text-embedding-ade
MТEB
62.26
60.99
85.53
85.45
LoCo
82.4
52.7
54.16
51.9
JinaLC
58.2
55.25
50
55
60
65
70
75
80
85
Figure 1: Benchmarking Text Embedding Model. Aggregate performance of nomic-embed--exx-yy OpenAl fext-embedding-adde OpenAl text-embeddinng--nnne 3-sslal and sina-embeddingg-eee base-v2 on both short and long context benchmarks. nomic embed is the only fully auditable long context model that excoods OpenAT lexi-embedding-ade and OpenAl text-embeedding---anne forms similarly outperforms Jina Base V2 all tasks. X-axis units vary per benchmark suite.
MTEB and LoCo. nomic embed per-
text- text-embedding-- --seeem-eeddinn----aaaal performance on short context (MTEB) and long context benchmarks (LoCo) (Table
Further, we are the first to release all training artifacts needed to train a high-performing text embedding model. We release the model weights, training code, and training data enable end-to-end auditability and replication of the model
2
Related Work
Text embedding models have historically been trained with sequence lengths less than or equal to 512 tokens. Recently, Günther et al. (2024) trained a long context text embedding model, jina-embeddinngs base-v2, but underperforms closed source text embedding models like text-embedding-adda MTEB benchmark as well the Jina Long Context Benchmark. Additionally, jina-embeddings- sinedddssns----e22 base-v2
on both the
underperforms other apintweight short-context text embedding models like E5 (Wang et al. 2022 GTE
[Li et al. 2023). and BGE (Xiao et al. 2023
Further, there is a lack of transparency into the training pipeline for high performing open-weight text embedding models. Many of these released models omit key details such as data source, data curation techniques, and training code. WWang et al. (2022) outlined their data filtering procedure for which includes first training model over a large noisy datasct and then using the resulting model filter low quality lext pairs. However, they do not release details on the model used for consistency filtering, how it was trained, or what data was used. They additionally do not release any training code data for the released embedding model. Similarly, Li et al. (2023) and Günther et al. (2024) did not detail the data sources for contrastive pretraining, omit details on how data filtering and mining was approached, and did not release training code.
Additionally, few details have been relcased on how closed source text embedding models are trained like Voyage-lite-O--nsstnie (Voyage 2023) and OpenAl's dext-embeddng--da-0000 and text-embedding-3 NNec. lakantan al. 2022
2

---

Published in Transactions on Machine Learning Research (02/2025)
3 Background
State-of-the-art text embedding models are generally trained in three stages: masked language modeling (Devlin et al. 2019), weakly-supervised contrastive pretraining, and contrastive finetuning (Wang et al. 2022
Li et al. 2023 Günther et al. 2023: 2024). Traditionally, finetuning involved leveraging labeled datasets such as MSMarco and SNLI Bowman al. 2015) to generate paired training data for the contrastive signal. Examples include SBERT (Reimers & Gurevych, 2019), SimCSE (Gao et al., 2022). and SGPT [Muennihhoff 2022 Recent models such as E5 (Wang et al. 2022). GTE (Li et al.
2023 BGE (Xiao
et al. 2023), InstructOR (Su al., 2023a), and Jina (Günther el al. 2023 2024 utilize mulli-stage regime in which a pretrained transformer is first contrastively trained using large corpus of weakly paired data (e.g. Quora, Reddit Comments) and then additionally finetuned on small, higher quality labeled datasets such as MSMarco. The two-stage paradigm significantly improves model quality as weakly paired data is available much greater quantity.
Table I: nomic-embed-text-wv/ is the only open-source long-context model to outperform closed source models like text-cmbedding-ala-.00 and text-embedd-3-sman on the short-context MTEB benchmark and the long context LoCo benchmark.
Model
Params
Seq
MTEB
LoCo
Jina LC
Weights
Code
Data
nomic-embesltext--1 nomi--enbedd-exttyaaabbeee
137M
8192
62.39
85.53
54.16
Yes
Yes
Yes
137M
8192
61.36
86.89
53.53
Yes
Yes
Yes
jins-embeeddinggs-bss--222
137M
8192
60.39
85.45
51.90
Yes
No
No
text-embedding-ada.
N/A
8192
60.99
52.70
55.25
No
No
No
text-ombeddingg--small
N/A
8192
62.26
82.4
58.21
No
No
No
E5-Mistral- 7b-iirral-7-- lext-embeddiigg--Bll
7B
4096
66.6
87.8
N/A
Yes
No
No
N/A
8192
64.59
79.4
58.69
No
No
No
3.1 Masked Language Modeling
Masked language modeling masks percentage of inputs and trains bidirectional transformer to predict the masked tokens (Devlin et al. 2019). The original BERT model was additional trained with binary auxiliary Next-Sentence Prediction (NSP) task. Liu et al.
2019 released RoBERTa in which they attained better per- formance by training on more data and for longer. They additionally removed the NSP task as it didn't show
any performance improvements. More recently, Portes et al. (2023) introduced MosaicBERT, an efficient and high performing BERT training recipe by increasing the masking rate, incorporating FlashAttention
(Dao et al. 2022 and other training optimizations.
3.2 Seakky-supervisee Contrastive Pretraining
Weakly-supervised contrastive pretraining aims to teach a model to distinguish the most similar documents from other irrelevant documents. To do so, we employ the InfoNCE contrastive loss (van den Oord et al. 2019). For given batch B (qo, do), (q1. di), ..., (gn, dn), we minimize the loss function:
Lc co=
Σ log (qg.d)/ + Σin (l,d)/ n
1
(1)
where s(g, d) is the (learned) score between query g and document We set to cosine similarity for all of our experiments. Contrary other approaches, we adopt unidirectional contrastive loss from query document. Other approaches like Günther et al. (2023) use a bidirectional contrastive loss by including the contrastive loss from document to query as well.
3

---

Published in Transactions on Machine Learning Research (02/2025)
3.3 Contrastive Finetuning
The last. stage of Iraining aims to boost. performance by utilizing human-labeled datasets. Several papers including Ni et al. (2021a b): Wang et al. (2022) Li et al. (2023) have shown that finetuning on these datasets leads to improvements in downstream performance, especially for QA and web-search retricval tasks. We adapt Equation to include hard negative documents in each batch:
Lc = -o=-->o log eslnddilt + Ej#t eduud)) + + (l
(2)
Here, we modify the partition function of the contrastive loss to include H hard negative documents dAn (1, m) which are documents specially chosen be close d but not true positive documents of q.
3.4 Rotary Positional Embeddings
Rotary Positional Embeddings (RoPE) are an alternative positional encoding introduced in (Su et al. 2023b)
that encode relative positional information through rotations within the attention-layers.
Following the notation in (Su et al. 2023b), setting 0 10, 000 is standard. We set A 1, 1,000 for experiments but found little to no per performance degradation.
3.5
RoPE Context Length Extrapolation
However, limitation with RoPE is in scaling to sequence lengths longer than during training. We discuss two methods: position interpolation and froquency-basod scaling.
3.5.1 Position Interpolation
Chen et al.
independently proposed extrapolating RoPE based models by interpolating the position indices to be within the original training sequence length. Following the notation in (Su et al. 2023b), given pretrained model with context length L, the position embedding function fw is modified as:
2023 and kaiokendev 2023
(wm m, 0a) fw (zm: mL
9a)
where L' > L is the target extended context length. This approach, while simple, requires fine-tuning on a smaller dataset to achieve stable performance at longer contexts.
3.5.2 Frequency-based Scaling
-WKrAware Scaling bloc97 2023 first proposed scaling the high frequencies more and low frequencies less by changing the base 0 in order to be 'NTK-aware" as it was shown in Tancik et al. (2020) that neural networks struggle to represent high frequencies well. The base is scaled by the ratio of the longer sequence length and the trained sequence length:
b' b 10/2
(4)
where Z.
Dynamic NTK Scaling Dynamic NTK scaling (emozilla aware by introducing hyperparameter to Equation
2023
Peng et al. 2023 improves upon NTK-
b' b ((a * 5) - (a 1)) -
P
(5)
This maintains the original position embeddings for sequences within the pretrained context length (lourrent S L) and gradually scales the embeddings as sequences grow longer, preventing abrupt performance degrada- tion. Additionally, this approach be used without any finetuning shown in (Peng et al. 2023
4

---

Published in Transactions on Machine Learning Research (02/2025)
4 Methods
4.1 Masked Language Modeling
4.1.1 Data
Following Devlin et al. (2019) we use BooksCorpus (Zhu et al. 2015) and Wikipedia dump from 2023 to train long context BERT model, hereinafter called nomic-bert-2048. Each document from BooksCorpus and Wikipedia is tokenized using the bert-base-uncased tokenizer from Devlin et al. (2019) and packed across documents to chunks of 2048 tokens. If document is shorter than 2048 tokens, we append another document until it fits 2048 tokens. If a document is greater than 2048 tokens, we split it across multiple documents. nomic-bert-2 2048 follows a similar training pipeline for masked language modeling as Portes et al. (2023). We omit next sentence prediction similarly Liu et al. (2019) and Portes et al. (2023 as it was shown to not improve performance and simplifies the training recipe.
4.1.2 Training Modifications
To train a long sequence length and efficient BERT, we adapt the BERT architecture. We make the following architecture changes to BERT base (Devlin et al. 2019):
Substituting absolute positional embeddings for rotary positional embeddings
Su ct al.
2023b
Using SwiGL.U activation instead of GeLU
Shazeer 2020
Using Flash Attention [Do et al. 2022
Setting Dropout to 0 (Geiping Goldstein
2022
Vocab size a multiple of 64 (Portss et al. 2023
Shoeybi et al.
2020
resulting in 137 million paramcter encoder.
We train all stages with max sequence length of 2048 and employ Dynamic NTK interpolation at inference
to scale to 8192 sequence length (Peng et al. 2023 emozilla 2003).. Additionally, we opt for SwiGLU versus
GeGLU like proposed in Portes et al. 2023 as runtime is roughly 25% faster for SwiGLU using the Flash Attention repository
(023) and remove the Next Sentence Prediction task to simplify the training recipe (Liu al., 2019 Portes al., 2023). We use the AdamW optimizer (Loshchilov & Hutter 2019) with max learning rate of 5e-4 with 81 0.9 82 0.98. We employ linear warmup of 6% of the total training steps and linear decay to 0. We use global batch size of 4096 with gradient accumulation over 8 batches. We utilize DeepSpeed (Rajbbandari et al. 2020) stage to larger batches into memory. Additionally, we use bfloat16 mixed precision and fp32 for gradient accumulation dtype. We disable gradient clipping (Liu et al., 2019) and set weight decay to 1e-5. We call our final model nomic-bert-2048 and also relcase its weights.
We use 30% masking rate instead of 15% following Portes
4.2
Contrastive Pretraining
4.2.1 Data
Similar to Wang et al. 20022
Li et al. 2023 Xiao et al. (2023 Ni et al. (2022), we use large collections of
publicly available data to form contrastive pairs. These datasets span various objectives and domains, from web retrieval to clustering of scientific articles. In total, we curated 470 million pairs across 29 dataset
Consistency Filtering: Since many of these datasets may contain noisy examples, we employ consistency
filtering remove the potential false positives in the dataset (Günther et al. 2023 Wang et al. 2022

---

Published in Transactions on Machine Learning Research (02/2025)
Consistency filtering uses pretrained model to filter out potential noisy examples in effort to improve data quality and subsequently model quality. Additionally, reducing the total number of examples needed to train a high quality text embedding model can reduce the overall cost to train the text embedding model.
For each pair, described as (query, document), we embed the queries and documents separately. We sample million points from the dataset and for each query, we find the top-k (in this case 2) neighbors using cosine similarity. If document not in the top-k neighbors, we discard the example.
Günther et al. (2023) uses all-MiniLM-LL-vv_" sistency filtering. However, we found that it regularly discarded retrieval pairs that were true positives but had low lexical overlap. We instead utilized gte-base (Li et al.) 2023), a 109 million parameter model for consistency filtering After filtering, we end up with 235 million pairs. The full dataset distribution can seen in Appendix
22 million parameter sentence embedding model for con-
We additionally explored consistency filtering using a cosine similarity threshold instead of the method described above. For a given pair, if the cosine similarity was greater equal to the threshold, kept the pair and otherwise discarded. However, we abandoned this approach in favor of top-k consistency filtering as we found through manual inspection the threshold consistency filtering discarded high quality retrieval pairs that had low cosine similarity. We additionally noticed lower retrieval scores in models trained using a threshold for consistency filtering versus using top-k consistency filtering
Curating Long Context Text Pairs: As the majority of these datasets are composed of sequences shorter than 2048 tokens we additionally curate long context datasets allow for the learning of long-range dependencies. We use Wikipedia titles paired with the corresponding body and S20RC (Lo et al. 2020) abstracts and full paper text from single paper.
Table 2: nomic-bert-2000 performs similarly to other short and long-context encoders when evaluated the GLUE benchmark.
Model
Seq
Bsz
Steps
Cola
SST2
MRPCSTSB QQP
MNLI
QNLI
RTE
Avg
MosaicBERT
128
4k
178k
0.59
0.94
0.89
0.90
0.92
0.86
0.91
0.83
0.85
JinaBERTBass
512
100k
0.51
0.95
0.88
0.90
0.81
0.86
0.92
0.79
0.83
SahertaBase
512
500k
0.64
0.95
0.90
0.91
0.92
0.88
0.93
0.79
0.86
MosaicBERT
2k
4k
70k
0.54
0.93
0.87
0.90
0.92
0.86
0.92
0.82
0.85
nomie-bert-2048
2k
4k
100k
0.50
0.93
0.88
0.90
0.92
0.86
0.92
0.82
0.84
You access the training data of nomic-embed-text-vl by visiting the code repository http:///mmmmiiitttttccoooooo....
You can explore
5M sample of our contrastive training pairs at
4.2.2 Training Modifications
We initialize the model for weakly-superriseed contrastive training with the weights of nomic-bert-2044. We use global batch size of 16,384. We use AdamW with a learning rate of 2e-4, B1 0.9, 88 = 0.999, and weight docay of 0.01. Gradient clipping is set to 1.0. We use lincar warmup schedule of 700 steps and an inverse square rool decay schedule.
We sample one data source and fill each batch with only data from that source to discourage the model learning sourcc-spocific shortcuts. We train with a max sequence length of 2048 for 1 full epoch over the weakly-supervised rastivvv data. Full details data composition can be found in Appendix B
Due to GPU memory constraints, employ GradCache (Lyyu Gao & Callan 2021 well mixed
precision training (Micikevicius et al. 2018
3 3all-MiniLM-L6-22 model bitwe wwwww// w.r. wwwetttprrrrrreeeeoooooooo htw. 4 bate-base model hergge//higgnngg inggane ww/thar www.t//egwwrggga....o. parigee-

---

Published in Transactions on Machine Learning Research (02/2025)
Finally, use task specific prefixes to break the symmetry of the biencoder in Wang et al (2022) Without prefixes, the model receives conflicting reward signal. Consider the case of determining which document is closest to the query "What is the capital of France?":
1. "What is the name of the capital city of France?
2. "Paris is the capital of France.
A semantic similarity task would consider the first closest, while question answering task would consider the second closest. Prefixes enable the model to distinguish between the behaviors specified by each of these tasks.
We use the following task-specific prefixes:
search_quee querr
@earch_documae
classification
clustering
inspired by Reimers al. (023)). ). We first break prefixes into categories: symmetric, where the query and document have similar structure, and asymmetric, where the query is usually single sentence and the document can be many sentences (Su et al. 2023ab. The first two prefixes are used for retrieval tasks: where search_quees query is used for the question and search document is used for the response. classification search_document
is used for STS-related lasks like rephrasals. clustering is used for tasks where the objective is to group semanticalll similar texts close together, like Arxiv title-abstract pairs. For symmetric tasks, the same prefix is appended to both the query and document.
4.2.3
Supervised Contrastive finetuning
4.2.4
Data
Supervised fine tuning is performed on MSMarco
Bajaj et al. 2018 Wang et al.
2023a
NQ Karpukhin
let al. 2020
Gao & Callan 2021). ) NLI (Gao et al. 2022), HotpotQA (Yang et al.
2018)
FEVER Thorne
et al.
2018
portions MEDI Su et al. 2023a), WikiAnswers (Fader et al. 2014 and Reddi
For the datasets MSMarco, NQ, NLI, FEVER, and HotpotQA, we train over the released training sets from the BEIR benchmark (Thakur al. 2021 For the retrieval datasets (MSMarco, NQ, HotpotQA, and Fever), mine negatives, if not already mined, using gte-base (Li et al., 2023 For each (g. d) pair, we find the top 20 documents among the corpus most similar to the query y, excluding d, and use these hard negatives. For other non-retrieval datasets, we randomly sample negatives among the corpus in place of mining hard negatives as wo found that mining did not improve performance.
Although the BEIR component of MTEB was originally intended as a zero shot benchmark, several open source models, such as those in Xiao al. (2023); Li et al. (2023); Wang al. (2023b), report training train splits of BEIR benchmark datascts such as FEVER and HotpotQA. We report results for nomic- trained without FEVER, HotpolQA, and MEDI.
Similarly to the weakly supervised contrastive stage, we sample a dataset and fill a batch with all points from that chosen datasct. In total, we train on 1.6 million datapoints. The full dataset distribution can be seen in Table 3

---

Published in Transactions on Machine Learning Research (02/2025)
Table 3: Supervised finetuning dataset distribution.
Dataset
Number of Samples
MSMarco
484,864
NLI
275,200
Reddit
199,680
Medi Supernli
177,408
Hotpot
169,728
Fever
139,776
Medi Stackexchange
100,352
NQ
69,888
Medi Flickr
50,944
Medi Wiki
24,832
4.2.5 Training Modifications
We train for one epoch using seven hard negatives per pair and a batch size of 256. We employ a learning rate of 20-5, 31 = 0.9, B2 = 0.999, and weight decay of 0.01. Gradient clipping is set to 1.0. We use a linear warmup schedule of 400 sleps and a linear cooldown and train with prefixes as described above. We found that increasing the number of negatives above does not significantly improve performance. We also found that training for multiple epochs hurts performance. Instead of choosing the first N negatives, randomly sampled the mined negatives. We found this to improve performance some of the mined negatives introduced false negatives.
5 Results
5.1
nomic-bert-2048 GLUE Results
We first evaluate nomic-bert-2048 on the GLUE benchmark BERT architecture performs similarly or better compared to similar encoders. The GLUE benchmark consists of 9 tasks, but we evaluate on similar to Liu et al. (2019). We follow the evaluation methodology presented Liu al. (2019). Roberta numbers are taken from Table 8 (Liu al.] 2019). MosaicBert numbers are taken from Table S1 in Portes et al. (2023) except for the 2048 model which we evaluated in the same manner nomic-bert-2048. BinaBertBBse Glue Test numbers reported in Table from (Günther et al. 2024
[Wang et al. 2019 to verify that our adapted
For each task, we train for 10 epochs with batch sizes 16, 32 and learning rate 1e-5, 2e-5, 3c-5 with a linear warmup of 6% across 5 seeds. The median score per task at the end of the 10 epochs is presented in Table
Note we report accuracy for MRPC and QQP and Pearson for STSB 0 Similar to Liu et al. 2019 we initialize from an MNLI checkpoint for RTE, STSB, and MRPC.
Across all tasks, nomic-bert-2045 scores similarly to MosaicBERT Poorts al. 2023 except Cola. MosaicBERT is trained with more gradient updates on C4 (Raffel et al. 2019). However, nomic-bert-20.4 was trained with longer sequence length and in effect has seen more tokens during pretraining. The difference in results could be due to a few reasons. First the training corpus could lead to better results on Cola as nomic-bert-2048 trains on Wikipodia and Bookscorpus while MosaicBERT trains on C4 which tends to skew towards shorter sequence lengths. Additionally, nomic-bert-204s utilizes RoPE while MosaicBERT uses ALiBi for long- ooggcontext extrapolation.
JinaBERT also trains similar model to MosaicBERT using ALiBI for long-context extrapolation and C4 its training corpus but sets the max sequence length to 512. It performs slightly worse average to nomic- bert-2048 and MosaicBERT. Even though it was trained similarly to MosaicBERT,, JinaBERT performs worse on Cola, RTE, and QQP. These findings are similar when compared to nomic-bert-2048 except Cola where JinaBERT outperforms nomic-bert-2048.
571729519

---

Published in Transactions on Machine Learning Research (02/2025)
Table MTEB benchmark results (Muennighof al, 2023).
nomic-embed--ext-v/ outperforms similarly
sized models on short-context tasks except DGE-BBase
Category +
Params.
Cls.
Clust.
PairCls.Rerani Retr.
STS
Summ. Avg
Number of datasets >>
12
3
56
Unsupervised Models
Glove Pennington et al. 2014
0.3B
57.3
27.7
70.9
43.3
21.6
61.9
28.9
42.0
SimCSE (Gao et al. 2022
110M
62.5
29.0
70.3
46.5
20.3
74.3
31.2
45.5
nomic-ombedd-text- unsup
137M
71.2
42.5
83.7
55.0
48.0
80.8
30.7
59.9
Supervised Models
Sim
Gao al.
2022
110M
67.3
33.4
73.7
47.5
21.8
79.1
23.3
48.7
Contriever (zacard et al.
2022a
110M
66.7
41.1
82.5
53.1
41.9
76.5
30.4
56.0
E5baso
Wang ct al.
2022
110M
75.2
44.2
86.0
56.6
50.6
82.1
30.2
61.6
GTEpasa
Li et al. 2023)
110M
73.0
46.2
84.6
58.6
51.1
82.3
31.2
62.4
BGE
Xiao et al.
2023
110M
75.5
45.8
86.6
58.9
53.3
82.4
31.1
63.6
Jina,2 (Günther al.
2024
137M
73.5
41.7
85.4
57.0
47.9
80.7
31.6
60.4
nomie-embed-text-v// blated
137M
73.6
43.7
84.6
53.3
51.4
80.2
31.3
61.4
nomic-embed-lext-v/
137M
74.1
43.9
85.2
55.7
52.8
82.1
30.1
62.4
Fanarge-v2
Wang et al.
2022
335M
75.2
44.5
86.0
56.6
50.6
82.1
30.2
62.3
GTEiagg
Li et al. 2023
335M
73.3
46.8
85.0
59.1
52.2
83.4
31.7
63.1
BGEga
Xiao et al.
2023
335M
76.0
46.1
87.1
60.0
54.3
83.1
31.6
64.2
GTR
Ni et al.)
2021a
4.8B
67.4
42.4
86.1
56.7
48.5
78.4
30.6
59.0
Sentence- 15xx1
Ni et al.
2021b
4.8B
73.4
43.7
85.1
56.4
42.2
82.6
30.1
59.5
text-embedding-adae
NA
70.9
45.9
84.9
56.3
49.3
81.0
30.8
61.0
text-embedding-3--ae
NA
73.2
46.7
85.0
56.7
51.1
81.6
31.1
62.3
-ext-eebbddding--11aare
NA
75.5
49.0
85.7
59.2
55.4
81.7
29.9
64.6
E5nistral
(Wang et al. 2023b)
7B
78.5
50.3
88.3
60.2
56.9
84.6
31.4
66.6
5.2
Text Embedding Benchmark Results
To evaluate nomic-embed-text-vl effectiveness as a text encoder, we evaluate it on MTEB (Muennighoff
et al. 2023
Jina's 's Long Context Benchmark (Günther al. 2024), and LoCo (Saad-Paloon al. 2024).
5.2.1 MTEB Results
MTEB is general text embedding benchmark released by Muennighoff et al. (2023) It measures text embedding performance across tasks Classification, Clustering, Pair Classification, Reranking, Retrieval, STS, and Summarization.
During evaluation, we add the classification prefix to both the query and document for the Classification, Pair Classification, STS, and Summarization tasks. We add the clustering prefix both the query and document for the Clustering task. And we add the search, pearch_query prefix to the query and search, search_document document prefix to the document for the Retrieval lask. We truncate all texts to 512 tokens. Additionally, we found better performance by not L2 normalizing the embeddings for the Classification task, similar to the evaluation code released by Wang et al. (022). For all other tasks, we L.2 normalize the embeddings.
The performance of nomic-embedd-lext-y and saaii--ennbe--exxt----aallll are broken down by task in Table Compared to similarly sized open-source text embedding models, nomic-embed- text--ll outperforms all models except BGE-base (Xiao et al.] 2023). Additionally, nomic-embed-teet outperforms larger open- source text embedding models like E5 Large (Wang al. 2022), GTR XXL (Ni al.) (021a). and Sentence T5 XXT. (Ni et al. 2021b)
9

---

Published in Transactions on Machine Learning Research (02/2025)
Table 5: Jina Long Context benchmark results. nomic-embed-text--V outperforms jina-embeeddinng--22baea and performs similarly to sust-cnbbedding-ada--000
Model
Seq
NarrativoQA
WikiCities
SciFact
BigPatent
Avg
jine-eebbeddding----s---22
128
19.6
79.9
62.1
14.4
44.0
nomll--mbbad--eet----aaaattt
128
20.8
86.8
65.2
17.5
47.6
tomic-embed--eext--11
128
20.1
90.0
65.4
18.5
48.5
text-embbediing--aaa-02
128
25.4
84.9
68.8
16.6
48.9
texti-embeddinn----mmall
128
29.5
87.5
68.8
15.0
50.2
text--enbboddinn----lare
128
45.6
87.9
74.8
16.5
56.2
jina-enbbedinngg--bbee
512
21.3
79.3
66.7
21.9
47.3
512
25.7
81.9
71.5
23.7
50.7
nomic-embee--exxt-vv
512
23.9
88.7
70.5
25.3
52.1
text-embeeding--aa-000
512
25.5
84.8
72.6
23.0
51.5
text=-embeddiinn---mmall
512
32.2
89.0
73.2
23.6
54.5
text-embbediing---larg
512
48.1
89.9
77.6
23.6
59.6
jina-enbbediingg--aaee2
8191
39.4
75.7
69.4
23.1
51.9
nonio--abbaa--txx------bbbaaat
8191
44.0
77.4
69.1
23.6
53.5
nomic-embed--text-vv
8191
37.8
84.3
70.2
24.5
54.2
text-embbeding--daa-02
8191
41.1
84.7
72.7
22.5
55.3
text-embeddiing----malll Lext-embediigg--Laargg
8191
47.1
89.9
73.3
22.5
58.3
8191
51.6
86.2
77.7
19.3
58.7
nomnc--ember-text-vl the best-performing 100M parameter class unsupervised model. nomic-embed nomic-embed--ext-v competitive with the top-performing models in both the 7B parameter class and with models trained in supervised setting specifically for the LoCo benchmark.
Table 6: LoCo benchmark results (Saad-Falcon et al. 2024
Model
Seq
Param.
Tau Scr.
Tau Gov.
Tau QMS.
QASP.
QASP.
Avg Abs.
Tit.
Art.
Art.
M2-Bert (Saad-Falcon et al.
2024
2048
80M
81.8
94.7
58.5
87.3
95.5
83.6
Jin8pase-r2 Günther al. 2024
2048
137M
87.2
97.7
35.1
95.3
99.7
83.0
nomi--embel.-txx-----bbbaae
2048
137M
83.1
97.3
49.4
97.4
99.9
85.4
nomie-ombadltext-v/
2048
137M
86.1
96.9
47.8
96.1
99.7
85.3
nomic-embed-text--1
4096
137M
89.0
97.4
45.7
95.8
99.9
85.6
nomie-embell-exx--llallaa
4096
137M
89.1
97.6
49.6
97.5
99.9
86.7
Einistral Wang ct al.
2023b
4096
7B
95.9
98.3
46.8
98.4
99.8
87.8
M2-Bert
Saad--Flloon et al.
2024
8192
80M
94.7
96.5
64.1
86.8
97.5
87.9
Jinabase-r2
Günther et al.
2023
8192
137M
93.3
98.6
40.8
95.1
99.3
85.5
8192
137M
92.5
97.8
47.6
96.5
99.9
86.9
nomie-onbed-text--7
8192
137M
90.9
97.8
44.2
94.9
99.9
85.5
lext-embeediing-aca-0000
8192
N/A
37.3
44.3
7.30
85.1
89.7
52.7
text-embedding----nmee
8192
N/A
92.2
97.7
27.4
95.9
98.9
82.4
-ereeembeedinnn----aaee
8192
N/A
88.0
93.6
25.5
93.2
96.8
79.4
Compared to closed-source models, seaii--embe--eex---te outperforms text-eebeddiin-adaae ad020200 and text- enbedding-3-small on average and notably the Retrieval task. nomic-embod-texte is the only open-source long-context text embedding model outperform cext-eebedding-da- and text-ombeddinng- on MTEB.
10

---

Published in Transactions on Machine Learning Research (02/2025)
unsurprisingly performs worse than nomic-embed-test--/ and other open- source text embedding models that finetune on the training sets of BEIR like BGF-base and GTE-basee However, homi--embad--ext---aaaaaaaee
still outperforms text-embedding-addee v2 and is competitive with E5-base v2. jina-enbedding--aseee bass-d---i is trained on similar datascts to nomic- embedd-extt-llablles yet nomic-embedd-eext- --ablateed outperfoms jina-embeedding- baaeiigg--bb-ev base-v2 on MTEB.
and jina-embeddings- base-
5.2.2 Long Context Results
However, as noted in Günther et al. (2024), MTEB has very few datasets that include long sequences. To evaluate nomic-enbed-text-ve performance on longer sequences, we consider two additional benchmarks: the Jina Long Context Dataset (Günther et al. 2024 well the LoCo benchmark from Saad-FFlcon
cet al. (2024). Although nomic-embed-testt->> was trained with max sequence length of 2048, we are able to use length extrapolation techniques proposed in lemozilll (2023 Peng et al. (2023)_
For texts longer than 2048, the max sequence length nomic-embed-texxt->) was trained on, we employ Dy- namic NTK Interpolation described Equation We set to 2.
5.2.3 JinaAl Long Context Benchmark
The Jina Long Context Benchmark (Günther et al. 2024) evaluates datascts across Retrieval and Clustering; namely, NarrativeQA (Günther et al. 2024). WikiCites M SciFact (Wadden et al. 2020), and BigPatent (Sharma et al., 2019). Similar to Günther et al. (2024). we report the V-scores and NDCG@10 for the clustering and retrieval datasets respectively. We evaluate all models at sequence length 128, 512, and on NarrativeQARettiera and Scifact, we use the search_ search_querry query and search, document prefixes for the query and document. respectively. For AegPatentClustring and WikiCities, we use the clustering prefix for both the query and document. Results are presented in Table 5
8191. For nomic-embed-texxt->/ and nomic-embed- wemmee--eet----aallaeee
Numbers for text- aattt--mmedding-ddd--aeaa and jhaa-enbedings--ase--22 are taken from Günther et al. 2024
Across all context lengths, nomic-embed-text-vi outperforms jinaembbeddings---bba... When evaluated on shorter sequence lengths, nomic-embed-teext-/ performs similarly to sort-embedding-atke ada-002 but is outper- formed at 8k context. Additionally, nomic-embed-text-w vl-blate outperforms jina-embeddinns-v//. -2-bae,,,--. but underperforns nomic-embed--ext--ll.
However, nomic-embed-text-v/ underperforms eost-embeeddin--da--0000 text-embeedding---nnne and text- embedding-3-large. Without any information on training data or architecture for the closed-source models, it's unclear why the gap exists. It is also surprising to see text-embedding---larg performance decrease as sequence length increases from 512 to 8191 while performance increases for text-cmbeeddigg--smmal and lext-embeddiigg-002-00
Similar to results in Günther et al. 20233 we see lower performance in WikiCities across models as sequence length increases suggesting the task may not be good mcasure of long context embedding performance.
5.2.4 LoCo Benchmark
The LoCo Benchmark consists of 5 retrieval datasets: 3 datasets from Shaham et al. (002 and 2 from Dasigi et al. (2021). Similar to the other retrieval evaluations, we use the search_quert and search document prefixes for the query and document respectively. We evaluate nomic-embee--text->7 and jina-embeddings- baso-v2 at sequence length 2048, 4096, and 8192. We additionally include results from [Saad-Falcon et al.) (2024) as well even though the model was finetuned on training sets of these datascts. Results are presented in Table D We include the QASPER Abstract Articles dataset for completeness, but would like to highlight that many models seem to oversaturate the benchmark and may not be representative of long-context performance.
http:///mmgginaarrre///aastttttt wiki clusterin
http://haggighhcc..oodassssssss pacillg--ppt-ttetee clusterin
11

---

Published in Transactions on Machine Learning Research (02/2025)
At 2048 sequence length, nomic-embed-text-vl and nomi--embed--ext-----bbbe outperform jina- embeddings-v2-boss At 4096 sequence length nomic-embed-lextte and nomic-embeed-eex---1ie 1--ablated is able to perform similarly to E5 Mistral, a model 2 70x bigger on all tasks except Tau Scrolls.
Both nomic-embed-text-wy variants outperform gett-embedding-ada---000 and text-embeedingg- 3-all and perform similarly to jin-eebbeddings-2- at 8192 sequence length. Interestingly, nomie-embed-text-vll ablated outperforms nomic-embed-text- and tina-embedding-bae--e... base-v2 suggesting that the BEIR training data may be orthogonal to the LoCo tasks.
6 Conclusion
We release the first fully open-source long context text embedding model that surpasses OpenAl's text- embedding-Ada-PPe and Doxt-embeddiing-0---mmmm
performance on both sort and long context benchmarks. We release the model weights and training code under a permissible license as well as the recipe, including data, to reproduce the model. As of this writing, nomic-embed has garnered over 14 million downloads on the Hugging Face model hub, underscoring the widespread demand for performant open source model recipes.
References
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehension dataset, 2018.
bloc97. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation, 2023. URL htttps://www http://wwwwww.dttt/m//// reddit. com/r/LocalLaMM/ llama_models
comments/44127/ http:///tttttaaeeeeeeeeee rowo_allleeeeeo_eeeeeaae allows 11ama models, to ro_eave/
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Melhods in Natural Language. Processing (EMNTP). Association for Computational Linguistics, 2015.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. URL
wwwpgrgggg....o..oeerre 15595
William Coster and David Kanchak. Simple English Wikipedia: new text simplification task. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguislies: Human Language Technologies, pp. 665 669, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https::/aclanthooooy// otgp:0/100...
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness, 2022.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of informatiion-seeking questions and answers anchored in research papers. 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
emozilla. Dynamically scaled rope further increases performansceerreeeeooooo of long context llama with zero fine-tuning, 2023. URL http://ce..aa...rro.oooooo/////a httpa///aaooooaaaaaaaaaaaaaaa/aa further increases/
poopee
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. Open Question Answering Over Curated and Extracted Knowledge Bases. In KDD, 2014.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. EL15: long form question answering. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 3558- 3567. Association for Computational Linguistics, 2019. CRL httpt:tttt.i.orrre/eeeooo 10160
doi: 20.1863//11p19-111
Katja Filippora and Yasemin Altun. Overcoming the lack of parallel data in sentence compression. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.), Pro- ceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1481-1491, Seattle, Washington, USA, October 2013. Association for Computational Linguisties. URL https:
Hlaclanthologyoorg/D.....
Wikimedia Foundation. Wikimedia downloads. URL httpss//ddummps.iiiiieeeaaa
Luyu Gao and Jamie Callan, Condenser: a pre-training architecture for dense retrieval, 2021.
Tianyu Gao, Xingcheng Yao, and Dançi Chen. Simese: Simple contrastive learning of sentence embeddings, 2022.
Jonas Geiping and Tom Goldstein. Cramming: Training a language model on single gpu in one day, 2022.
Mansi Gupta, Nitish Kulkarni, Raghuveer Chanda, Anirudha Rayasam, and Zachary Lipton. Amazonqa:
A review-based question answering task, 2019.
Michael Günther, Louis Milliken, Jonathan Geuter, Georgios Mastrapas, Bo Wang, and Han Xiao. Jina embeddings: A novel set of high-performance sentence embedding models, 2023.
Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, and Han Xiao. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents, 2024.
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Information Science pp. 218-223, March 2017. doi: 10.5281 (5528/2eenode /eee/ezooo040 412036..
Christopher Hidey and Kathy McKeown. Identifying causal relations using parallel Wikipedia articles. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1424 1433, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P166 11355
1135. URL https:///ccaatthology/
org/P16- 1135
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. CodeSearchNett challenge: Evaluating the state of semantic code search. arXiv preprint XXic:1999.09436. 2019.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, 2022a.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models, 2022b.
kaiokendev. Things I'm learning while training superhot., 2023. URI. https: hwwpaaaooooeeeeeeitii.bbeee 10/ cillferteeddngggcoteee------o
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen lau Yih. Dense passage retrieval for open-domain question answering, 2020.
Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannanch Hajishirzi, and Chris Callison- Burch.. Gooag: Open question answering with diverse answer types, 2021.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Mahnaz Koupace and William Yang Wang. Wikihow: A large scale text summarization dataset, 2018.
Patrick Lewis, Ethan Perez, Alcksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval- augmented generation for knowledge-intensive nlp tasks, 2021a.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. Paq: 65 million probably-askod questions and what you can do with them, 2021b.
Zchan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Mcishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeltlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Dan S. Weld. S2ore: The semantic scholar open research corpus, 2020.
lya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Jiawci Han Luyn Gao, Yunyi Zhang and Jamic Callan. Scaling deep contrastive learning batch size under memory limited setup. In Proceedings of the 6th Workshop on Representation Learning for NLP, 2021.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training, 2018.
Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search, 2022.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding bench- mark, 2023.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidocke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. Text and code embeddings by contrastive pre-training, 2022.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint. Conference on Natural Language Processing (FMNIP-IJCNIPP), pp. 188-197, Hong Kong, China. November 2019. Association for Computational Linguistics. doi: 10.18653/v11D11--0
URL https:
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming- Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers, 2021a.
Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models, 2021b.
Jianmo Ni, Gustavo Heruandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 1864-1874, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 146. URL
wttp:/thhhhooooooo//2222222 findings-acl..... acl.1116sa...166
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word repre- sentation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532 1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/DD441122. URL httpp://aclaathhology http://aaanniioooooooo///------- org/D14-1162
Jacob Portes, Alex Trott, Sam Havens, Daniel King, Abhinav Venigalla, Moin Nadeem, Nikhil Sardana, Daya Khudia, and Jonathan Frankle. Mosaicbert: A bidircctional encoder optimized for fast pretraining, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv e-prints, 2019.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprchension of Text. arXiv e-prints, art. arXiv:1606.05250, 2016.
Ori Ram, Yoar Levine, Itay Dalmedigos, Dor Mublgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmentte language models, 2023.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks, 2019.
Nils Reimers, Elliot Choi, Amr Kayid, Alckhya Nandula, Manoj Govindassamy, and Abdullah Elkady. Introducing embed v3, Nov 2023. URL https://tttttt../: http::/://txt.ccoeeeeeee cohere. com
Jon Saad-Falcon, Dan Fu, and Simran Arora. Long-context retrieval models with monarch mixer, Jan 2024. URL httts::/hazyreeeaatt stanford. b2-
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10.1865//11/P17-0999 URL https://www aclweb. prrg/antholog//
P199
Uri Shaham, Elad Segal, Maor Tvgi, Avia Efrat. Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. SCROLLS: Standardized CompaRison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 12007-12021, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL htttps://cclatthologg
org/2022 2022. smnnp-mmin..8l 823
Eva Sharma, Chen Li, and Lu Wang. BIGPATENT: A large-scale dataset for abstractive and coherent summarization. CoRR, hts//996.037441 2019. URL http://arxiv... http:://arivv.rr//ab//1o000
Noam Shazeer. Glu variants improve transformer, 2020.
Mohammad Shocybi, Mostofa. Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatrou-Im: Training multi-billion parameter language models using model parallelism, 2020.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: insrruction-inettunee text embeddings, 2023a.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023b.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains, 2020. URL
10739
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A het- erogenous benchmark for zero-shol evaluation of information retrieval models, 2021.
James Thore, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In NAACL-HLT, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019.
Voyage. Excited to announce voyage embeddings!, Nov 2023. URL 29/vo voyagoyyeeerat emm.gdinggm
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Han- nanch Hajishirzi. Fact or fiction: Verifying scientific claims. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 7534-7550, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1 http://1112200..mmpmmmmmaa
URL httips://aclanthologg org/ emnlp-main.600
Alex Wang, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019. In the Proceedings of ICLR.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, 2022.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjum Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Simlm: Pre-training with representation bottleneck for dense passage retrieval, 2023a.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furn Wei. Improving text embeddings with large language models, 2023b.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification, 2016.
Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, 2015.
A Training Resources
Full training of nomic-embed--text-v/ can be conducted in a single week on one 8xH100 node, Masked language modcling of nomic-bert-2048 takes roughly 4 days. Contrastive pretraining lasts 3 and a half days. Contrastive finetuning takes one hour. We encourage the reader initialize from our samic-bert-2045 Unsupervised Constrastive checkpoints, released under the same license nomic-embed-text-v..
16

---

Published in Transactions on Machine Learning Research (02/2025)
Table 7: Weakly Unsupervised Dataset Distribution
Dataset
Datapoints
% Dataset
Reddi
64,978,944
0.28
PAQ Lewis et al.
2021b
52,953,088 088
0.23
Amazon Reviews Ni et al. 2019
38,682,624
0.16
S2ORC Title Abstract [L ct al.
2020
35438592
0.15
WikiAnswers (Fader et al. 2014
9,912,320
0.04
S2ORC Citation Titles Lo et al.
2020
7,585,792
0.03
S2ORC Abstract Citation
2020
7,503,872
0.03
S2ORC Abstract Body (Lo
2020
6,389,760
0.03
Wikipedia Title Body Foundation)
6,078,464
0.03
Gooaq (Khashabi et al.
2021
1,245, 184
0.01
Codesearch (Husain ct al.
2019
835,584
<.01
AGNews Zhhng et al.
2016
201
409,600
<.01
CCNews Hamborg et al.
344,064
<.01
NPR
344,064
<.01
CNN See et al. 2017
278,528
<.01
Yahoo Title-Answer
262,144
<.0
AmazonQA Gupta ct al
2019
212,992
<.0
Yahoo Title-Queste
196,608
<.01
Sentence Compression Filippova & Altun
2013
163,840
<.01
YahooQA
131,072
<.0
EL15 Fan et al. 2019
98,304
<.01
Altlex (Hidey & McKeown 2016
98,304
<.01
Wikihow (Koupaee & Wang 2018
81,920
<.01
SimpleWiki (Coster& & Kauchak 2011
81,920
<.01
Stacknxchange Duplicate Questions
65,536
<.01
StackExchange Title Body
65,536
<.01
StackExchange Body Body
65,536
<.01
Quora Duplicate Questions
32,768
<.01
SQuAD RRajpurkar et al.
16,384
<.01
Total
234,553,344
https://www.ken
https:
https://www
quora. http///wwaa.aoaaaao.oooo//oaaaar h Question-Paira
B Pretraining Dataset Distribution
Weakly-supervised contrastive pretraining datasets are detailed Table 7 This is the number of datapoints per source after consistency filtering.
17


Reviews:
[{"review_id": "JUUShAfUkB", "reviewer": "Reviewer_FaWz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents nomic-embed-text-v1, an open-source English text embedding model with an 8192-token context length. Unlike typical sentence embedding models that build upon pretrained decoder-based language models, this work initiates from a masked language model, emphasizing data quality and quantity. The model demonstrates comparable performance on both short and long documents, contributing to advancements in representation models. Its relatively small parameter size is advantageous for applications like retrieval, where efficiency is crucial.\n\nWhile the paper is commendable, certain sections could benefit from refinement. For instance, Section 3.4 covers too much general knowledge that doesn’t directly reflect the authors’ contributions; condensing this part could make room for more discussion on novel findings, such as the impact of varying the parameter θ. Similarly, if Section 3.5.2’s discussion on NTK-Aware Scaling isn’t utilized in the experiments, it might be more appropriate for the supplementary materials. Additionally, enhancing table designs and descriptions would improve readability. In Table 2, reordering columns to place ‘Model’ and ‘Seq’ first could help readers quickly identify key differentiators. In Section 4.2.1, the raw version data uses words \"472 million\", the filter data uses words \"~235M pairs\". \n\nOverall, the paper makes a significant contribution, and addressing these suggestions could enhance its clarity and impact.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a comprehensive account of developing the nomic-embed-text-v1 model, an open-source English text embedding model with an 8192-token context length. The training process comprises three primary stages:\n1. Masked Language Modeling (MLM) Pretraining: The model, nomic-bert-2048, was pretrained using a masked language modeling objective on datasets such as BooksCorpus and Wikipedia.\n2. Unsupervised Contrastive Pretraining: Following MLM pretraining, the model underwent unsupervised contrastive learning using a large corpus of weakly paired data, including question-answer pairs from forums and title-body pairs from reviews.\n3. Supervised Fine-tuning: The final stage involved fine-tuning the model on smaller, high-quality labeled datasets, such as MSMarco, to enhance its performance on specific tasks.\n\nAdditionally, the authors provide detailed methodologies for data collection and filtering, ensuring the model’s reproducibility and performance. They have released the training code, model weights, and a curated dataset of 235 million text pairs under an Apache 2 license, facilitating full replication of the model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Except the weaknesses mentioned above, \nThe Consistency Filtering section is important but a little bit hard to follow, e.g. Offering brief descriptions of the all-MiniLM-L6-v2 and gte-base models to contextualize their selection.", "broader_impact_concerns": "The authors does not delve into potential ethical, societal implications associated with the deployment of the data and models.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pdj31saR2I", "reviewer": "Reviewer_nt2w", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper fully open-sources its implementation details, promoting transparency and fostering advancements in related research and applications.\n- Extensive experiments demonstrate that the proposed embedder achieves performance comparable to state-of-the-art models.\n- In long-context scenarios, the anon-embed-text-v1-ablated model outperforms various baseline models, showcasing its effectiveness in handling extended sequences.\n\nWeaknesses:\n- The model's performance on relatively short sequences falls short compared to baseline models.\n- With a relatively small number of parameters, the model shows clear disadvantages when compared to larger models such as E5.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This technical report presents Anon Embed, the first fully open and reproducible long-context embedder. It provides a comprehensive overview of the model architecture, training algorithm, and selected hyperparameters. The embedder is trained using a combination of masked language modeling, weakly supervised contrastive pretraining, and contrastive fine-tuning. To support scaling to 8192 tokens during inference, rotary positional embeddings and interpolation techniques are employed. Extensive experiments validate the model's performance, demonstrating its effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The contribution of the work could be enhanced by training and open-sourcing a larger model with improved performance.\n2. The presentation of result tables could be more organized and clear. For instance, the order of models and the usage of bolding is somewhat confusing, making it challenging to extract information efficiently.", "broader_impact_concerns": "The paper does not present any obvious concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4sOKtCcWT6", "reviewer": "Reviewer_2U5B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe paper does comprehensive experiments on training text embeddings, which covers from initial model pretraining, weakly supervised data and supervised data. It also performs broad evaluation on the most used benchmarks regarding different aspects of text embeddings.  The paper outputs a better BERT model with modifications from recent LLM updates in structure and surpass the performance of original BERT.\n\nThe paper also divides the contrastive learning into weakly and full supervised training with details about how the data was choosen and filtered, for example, consistency filtering and negative sample mining, achieving better performance with a relative smaller computation cost in inference.\n\nThe paper also promises code and data for this training pipeline, which iof cource benenfits the community.\n\n\nWeakness:\n\nThe paper leverages masked language model (training from scratch) as a base model for the text representation learning. I am concerned that the method may not be able to take advantage of the fast-pacing improvement from the generative text modeling. For example, this base model is limited to 2048 tokens in length, total training tokens, model size, etc. In the meantime, there has already been tiny but strong open-sourced gpt models which is trained on longer sentences, consumes more tokens and reasonable in model size. Unless there's a strong reason for BERT, I would suggest transferring to a tiny gpt as the base structure.\n\n\nQuestions:\n\nFor the data filtering, how did you decide the hyperparameters of negative mining and consistency filtering, is there any metric to determine this hyperparameters?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The key contributions of the paper are the introduction of anon-embed-text-v1, a fully reproducible, open-source long-context text embedding model with an 8192-token context length. The model outperforms OpenAI’s text-embedding-ada-002 and text-embedding-3-small on both short-context (MTEB) and long-context (LoCo, Jina Long Context Benchmark) benchmarks, despite having only 137M parameters. The paper provides complete transparency by releasing model weights, training code, and curated training data. The training pipeline includes masked language modeling, weakly-supervised contrastive pretraining, and supervised contrastive finetuning, utilizing techniques like consistency filtering and hard negative mining. Additionally, the model leverages architectural optimizations such as RoPE, SwiGLU, and FlashAttention for efficiency and scalability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "A few more details about the  negative mining and consistency filtering process, and the metric of choosing such filtering and mining hyperparameters. \n\nA few more details determining corpus size of the second and third stage and when you would think the stage2 is good enough for starting stage 3 training.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers suggestion several revisions, please consider it include them. The writing style of the tables should be also improved.
Decision: Accept (poster)