YuxinJiang
commited on
Commit
•
24aba6b
1
Parent(s):
ca631ad
Update README.md
Browse files
README.md
CHANGED
@@ -2,13 +2,14 @@
|
|
2 |
license: mit
|
3 |
---
|
4 |
# PromCSE: Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning
|
5 |
-
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/
|
|
|
6 |
arXiv link: https://arxiv.org/abs/2203.06875v2
|
7 |
Published in [**EMNLP 2022**](https://2022.emnlp.org/)
|
8 |
|
9 |
-
Our code is modified based on [SimCSE](https://github.com/princeton-nlp/SimCSE) and [P-tuning v2](https://github.com/THUDM/P-tuning-v2/). Here we would like to sincerely thank them for their excellent works.
|
10 |
|
11 |
-
We have released our supervised and unsupervised models on huggingface, which acquire **Top 1** results on 1 domain-shifted STS task and 4 standard STS tasks
|
12 |
|
13 |
[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deep-continuous-prompt-for-contrastive-1/semantic-textual-similarity-on-cxc)](https://paperswithcode.com/sota/semantic-textual-similarity-on-cxc?p=deep-continuous-prompt-for-contrastive-1)
|
14 |
|
@@ -34,6 +35,9 @@ We have released our supervised and unsupervised models on huggingface, which ac
|
|
34 |
|
35 |
If you have any questions, feel free to raise an issue.
|
36 |
|
|
|
|
|
|
|
37 |
|
38 |
## Setups
|
39 |
|
@@ -52,7 +56,7 @@ In the following section, we describe how to train a PromCSE model by using our
|
|
52 |
|
53 |
|
54 |
### Evaluation
|
55 |
-
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/
|
56 |
|
57 |
Our evaluation code for sentence embeddings is based on a modified version of [SentEval](https://github.com/facebookresearch/SentEval). It evaluates sentence embeddings on semantic textual similarity (STS) tasks and downstream transfer tasks. For STS tasks, our evaluation takes the "all" setting, and report Spearman's correlation. The STS tasks include seven standard STS tasks (STS12-16, STSB, SICK-R) and one domain-shifted STS task (CxC).
|
58 |
|
@@ -179,7 +183,13 @@ All our experiments are conducted on Nvidia 3090 GPUs.
|
|
179 |
|
180 |
|
181 |
## Usage
|
182 |
-
We provide
|
|
|
|
|
|
|
|
|
|
|
|
|
183 |
```bash
|
184 |
python tool.py \
|
185 |
--model_name_or_path YuxinJiang/unsup-promcse-bert-base-uncased \
|
@@ -271,4 +281,3 @@ Please cite our paper by:
|
|
271 |
pages = "3021--3035",
|
272 |
}
|
273 |
```
|
274 |
-
|
|
|
2 |
license: mit
|
3 |
---
|
4 |
# PromCSE: Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning
|
5 |
+
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1lanXViJzbmGM1bwm8AflNUKmrvDidg_3?usp=sharing)
|
6 |
+
|
7 |
arXiv link: https://arxiv.org/abs/2203.06875v2
|
8 |
Published in [**EMNLP 2022**](https://2022.emnlp.org/)
|
9 |
|
10 |
+
Our code is modified based on [SimCSE](https://github.com/princeton-nlp/SimCSE) and [P-tuning v2](https://github.com/THUDM/P-tuning-v2/). Here we would like to sincerely thank them for their excellent works. Our models acquires comparable results to [PromptBERT](https://github.com/kongds/Prompt-BERT) **without designing discrete prompts manually**.
|
11 |
|
12 |
+
We have released our supervised and unsupervised models on huggingface, which acquire **Top 1** results on 1 domain-shifted STS task and 4 standard STS tasks:
|
13 |
|
14 |
[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/deep-continuous-prompt-for-contrastive-1/semantic-textual-similarity-on-cxc)](https://paperswithcode.com/sota/semantic-textual-similarity-on-cxc?p=deep-continuous-prompt-for-contrastive-1)
|
15 |
|
|
|
35 |
|
36 |
If you have any questions, feel free to raise an issue.
|
37 |
|
38 |
+
[//]: <## Architecture>
|
39 |
+
[//]: <We add multi-layer trainable dense vectors as soft prompts to the input sequence, which means the input embeddings as well as each layer's hidden embeddings of prompts are optimized (the orange blocks). Note that all parameters of the pre-trained model are frozen (the blue blocks), thus reducing the number of tunable parameters to around **0.1\%**. The [CLS] token embedding of the last layer is selected as the sentence representation. The contrastive framework is the same as SimCSE.>
|
40 |
+
|
41 |
|
42 |
## Setups
|
43 |
|
|
|
56 |
|
57 |
|
58 |
### Evaluation
|
59 |
+
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1lanXViJzbmGM1bwm8AflNUKmrvDidg_3?usp=sharing)
|
60 |
|
61 |
Our evaluation code for sentence embeddings is based on a modified version of [SentEval](https://github.com/facebookresearch/SentEval). It evaluates sentence embeddings on semantic textual similarity (STS) tasks and downstream transfer tasks. For STS tasks, our evaluation takes the "all" setting, and report Spearman's correlation. The STS tasks include seven standard STS tasks (STS12-16, STSB, SICK-R) and one domain-shifted STS task (CxC).
|
62 |
|
|
|
183 |
|
184 |
|
185 |
## Usage
|
186 |
+
We provide [tool.py](https://github.com/YJiangcm/PromCSE/blob/master/tool.py) which contains the following functions (A quick start [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1lanXViJzbmGM1bwm8AflNUKmrvDidg_3?usp=sharing)):
|
187 |
+
|
188 |
+
**(1) encode sentences into embedding vectors;
|
189 |
+
(2) compute cosine simiarities between sentences;
|
190 |
+
(3) given queries, retrieval top-k semantically similar sentences for each query.**
|
191 |
+
|
192 |
+
You can have a try by runing
|
193 |
```bash
|
194 |
python tool.py \
|
195 |
--model_name_or_path YuxinJiang/unsup-promcse-bert-base-uncased \
|
|
|
281 |
pages = "3021--3035",
|
282 |
}
|
283 |
```
|
|