Update model with weights trained on deduplicated `The Pile` (#1)

- [update] Release model weights from deduped Pile training (04653c1139332fc0745f4c0290f25ccb0e8d090b)
- [fix] Make `Training Data` section header-2 (7664ba3e78aae9c3b90389cb3241fbcb8e9f0b69)

README.md

@@ -6,15 +6,18 @@ datasets:
 - pile
 metrics:
 - nDCG@10
+- MRR
 ---
 
 # Carptriever-1
 
-# Model description
+
+## Model description
 
 Carptriever-1 is a `bert-large-uncased` retrieval model trained with contrastive learning via a momentum contrastive (MoCo) mechanism following the work of G. Izacard et al. in ["Contriever: Unsupervised Dense Information Retrieval with Contrastive Learning"](https://arxiv.org/abs/2112.09118).
 
-# How to use
+
+## How to use
 
 ```python
 from transformers import AutoTokenizer, AutoModel
@@ -51,11 +54,13 @@ for sentence, score in sentence_score_pairs:
     print(f"\nSentence: {sentence}\nScore: {score:.4f}")
 ```
 
-# Training data
 
-Carptriever-1 is pre-trained on [The Pile](https://pile.eleuther.ai/), a large and diverse dataset created by EleutherAI for language model training.
+## Training data
+
+Carptriever-1 is pre-trained on a de-duplicated subset of [The Pile](https://pile.eleuther.ai/), a large and diverse dataset created by EleutherAI for language model training. This subset was created through a [Minhash LSH](http://ekzhu.com/datasketch/lsh.html) process using a threshold of `0.87`.
 
-# Training procedure
+
+## Training procedure
 
 The model was trained on 32 40GB A100 for approximately 100 hours with the following configurations:
 
@@ -73,24 +78,51 @@ The model was trained on 32 40GB A100 for approximately 100 hours with the follo
     - `momentum    = 0.999`
     - `temperature = 0.05` 
 
-# Evaluation results
 
-We provide evaluation results on the [BEIR: Benchmarking IR](https://github.com/beir-cellar/beir) suite.
+## Evaluation results
+
+#### [BEIR: Benchmarking IR](https://github.com/beir-cellar/beir)
 
-| nDCG@10                                  | Avg         | MSMARCO | TREC-Covid | NFCorpus | NaturalQuestions | HotpotQA | FiQA | ArguAna | Tóuche-2020 | Quora | CQAdupstack | DBPedia | Scidocs | Fever | Climate-fever | Scifact |
-|------------------------------------------|-------------|---------|------------|----------|------------------|----------|------|---------|-------------|-------|-------------|---------|---------|-------|---------------|---------|
-| Contriever*                               | 35.97 |    20.6 |       27.4 |     31.7 |             25.4 |     48.1 | 24.5 |    37.9 |        19.3 |  83.5 |        28.4 |    29.2 |    14.9 |  68.2 |          15.5 |    64.9 |
-| Carptriever-1 | 34.29 |   18.81 |       **46.5** |     28.9 |             21.1 |    39.01 | 20.2 |    33.4 |        17.3 |  80.6 |        25.4 |    23.6 |    14.9 |  59.6 |          **18.7** |    **66.4** |
+We report the following BEIR scores as measured in normalized discounted cumulative gain (nDCG@10):
 
-\* Results are taken from the Contriever [repository](https://github.com/facebookresearch/contriever).
+|   Model       | Avg   | MSMARCO | TREC-Covid | NFCorpus | NaturalQuestions | HotpotQA | FiQA | ArguAna | Tóuche-2020 | Quora | CQAdupstack | DBPedia | Scidocs | Fever | Climate-fever | Scifact  |
+|---------------|-------|---------|------------|----------|------------------|----------|------|---------|-------------|-------|-------------|---------|---------|-------|---------------|----------|
+| Contriever*   | 35.97 |    20.6 |       27.4 |     31.7 |             25.4 |     48.1 | 24.5 |    37.9 |        19.3 |  83.5 |       28.40 |    29.2 |    14.9 | 68.20 |          15.5 |    64.9  |
+| Carptriever-1 | 34.54 |   18.83 |   **52.2** |     28.5 |             21.1 |     39.4 | 23.2 |    31.7 |        15.2 |  81.3 |       26.88 |    25.4 |    14.2 | 57.36 |      **17.9** |    64.9  |
+
+\* Results are taken from the Contriever [GitHub repository](https://github.com/facebookresearch/contriever).
 
 Note that degradation in performance, relative to the Contriever model, was expected given the much broader diversity of our training dataset. We plan on addressing this in future updates with architectural improvements and view Carptriever-1 as our first iteration in the exploratory phase towards better language-embedding models.
 
-# Appreciation
 
-All compute was graciously provided by [Stability.ai](https://stability.ai/). 
+#### [CodeSearchNet Challenge Evaluating the State of Semantic Code Search](https://arxiv.org/pdf/1909.09436.pdf)
+
+We provide results on the CodeSearchNet benchmark, measured in Mean Reciprocal Rank (MRR), following the code search procedure outlined in Section 3.3 of Neelakantan et al.'s ["Text and Code Embeddings by Contrastive Pre-Training"](https://arxiv.org/pdf/2201.10005.pdf).
+
+`Candidate Size = 1,000`
+
+| Model           | Avg   | Python | Go    | Ruby  | PHP   | Java  | JS    |
+|-----------------|-------|--------|-------|-------|-------|-------|-------|
+| Carptriever-1   | 60.24 |  65.85 | 63.29 |  62.1 |  59.1 | 55.52 | 55.55 |
+| Contriever      | 49.39 |  54.81 |  58.9 | 55.19 | 38.46 | 44.89 | 44.09 |
+
+
+`Candidate Size = 10,000`
+
+| Model.          | Avg   | Python | Go    | Ruby  | PHP   | Java  | JS    |
+|-----------------|-------|--------|-------|-------|-------|-------|-------|
+| Carptriever-1   | 48.59 |  55.98 | 43.18 | 56.06 | 45.62 | 46.04 | 44.66 |
+| Contriever      |    37 |  45.43 | 36.08 | 48.07 | 25.59 | 32.89 | 31.44 |
+
+
+## Acknowledgements
+
+This work would not have been possible without the compute support of [Stability AI](https://stability.ai/).
+
+Thank you to Louis Castricato for research guidance and Reshinth Adithyan for creating the CodeSearchNet evaluation script.
+
 
-# Citations
+## Citations
 
 ```bibtex
 @misc{izacard2021contriever,

config.json

@@ -20,7 +20,7 @@
   "pooling": "average",
   "position_embedding_type": "absolute",
   "torch_dtype": "float32",
-  "transformers_version": "4.22.0.dev0",
+  "transformers_version": "4.21.3",
   "type_vocab_size": 2,
   "use_cache": true,
   "vocab_size": 30522

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dd49ac4881768f3554dd1df5f359421ad807d7643858de0bc06495b541ce8a71
+oid sha256:63f84de125685fd11067227632f0be12e124cfde031d4b47460e2220b8ee5f84
 size 1336496437