add files

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 384,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

2_Dense/config.json

@@ -0,0 +1 @@
+{"in_features": 384, "out_features": 128, "bias": true, "activation_function": "torch.nn.modules.activation.Tanh"}

2_Dense/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98a29078d83e39ae97f72162b048432ee1a0b2023d8cd12606c85a31742f8c29
+size 198183

README.md

@@ -1,3 +1,78 @@
 ---
 license: apache-2.0
 ---
+
+# HPD-MiniLM-F128
+
+This repository contains the pre-trained models for our paper [Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation](https://arxiv.org/abs/2203.07687).
+
+## Overview
+
+We propose **H**omomorphic **P**rojective **D**istillation (HPD) to learn compressed sentence embeddings. Our method augments a small Transformer encoder model with learnable projection layers to produce compact representations while mimicking a large pre-trained language model to retain the sentence representation quality.
+
+## Details
+
+This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 128 dimensional dense vector space and can be used for tasks like clustering or semantic search.
+
+The teacher model is [`princeton-nlp/sup-simcse-roberta-large`](https://huggingface.co/princeton-nlp/sup-simcse-bert-base-uncased) and the student model is [`nreimers/MiniLM-L6-H384-uncased`](https://huggingface.co/nreimers/MiniLM-L6-H384-uncased).
+
+## Usage
+
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+
+```
+pip install -U sentence-transformers
+```
+
+After installing the package, you can simply load our model
+```python
+from sentence_transformers import SentenceTransformer
+model = SentenceTransformer('Xuandong/HPD-MiniLM-F128')
+```
+
+Then you can use our model for **encoding sentences into embeddings**
+```python
+sentences = ['He plays guitar.', 'A street vendor is outside.']
+sentence_embeddings = model.encode(sentences)
+
+for sentence, embedding in zip(sentences, sentence_embeddings):
+    print("Sentence:", sentence)
+    print("Embedding:", embedding)
+    print("")
+```
+
+## Evaluation Results
+
+We evaluate our model on semantic textual similarity (STS) tasks. The results are:
+
+| STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R |  Avg. |
+|-------|-------|-------|-------|-------|--------------|-----------------|-------|
+| 74.94 | 84.52 | 80.25 | 84.87 | 81.90 |    84.98     |      81.15      | 81.80 |
+
+
+## Training
+
+Please refer to the github repo (https://github.com/XuandongZhao/HPD) for the details about the training.
+
+
+## Full Model Architecture
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
+  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
+  (2): Dense({'in_features': 384, 'out_features': 128, 'bias': True, 'activation_function': 'torch.nn.modules.activation.Tanh'})
+)
+```
+
+## Citation
+
+Please cite our paper if you use HPD in your work:
+
+```bibtex
+@article{zhao2022compressing,
+  title={Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation},
+  author={Zhao, Xuandong and Yu, Zhiguo and Wu, Ming and Li, Lei},
+  journal={arXiv preprint arXiv:2203.07687},
+  year={2022}
+}
+```

config.json

@@ -0,0 +1,25 @@
+{
+  "_name_or_path": "Xuandong/HPD-MiniLM-F128",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 384,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 6,
+  "num_hidden_layers": 6,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.11.3",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.0.0",
+    "transformers": "4.11.3",
+    "pytorch": "1.10.0+cu113"
+  }
+}

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Dense",
+    "type": "sentence_transformers.models.Dense"
+  }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50a0a61eff4ad95c33b9d69f1fbb3d0c6b3bc8d5d8e79dd554118abac8fc6847
+size 90896561

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "do_basic_tokenize": true, "never_split": null, "model_max_length": 512, "special_tokens_map_file": null, "name_or_path": "Xuandong/HPD-MiniLM-F128", "tokenizer_class": "BertTokenizer"}