aps6992 commited on
Commit
6d7f30f
1 Parent(s): aa590b5

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +150 -6
README.md CHANGED
@@ -29,15 +29,159 @@ from transformers import AutoAdapterModel
29
  model = AutoAdapterModel.from_pretrained("allenai/specter_plus_plus")
30
  adapter_name = model.load_adapter("allenai/spp_proximity", source="hf", set_active=True)
31
  ```
 
32
 
33
- ## Architecture & Training
34
 
35
- <!-- Add some description here -->
 
36
 
37
- ## Evaluation results
38
 
39
- <!-- Add some description here -->
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
 
41
- ## Citation
42
 
43
- <!-- Add some description here -->
 
29
  model = AutoAdapterModel.from_pretrained("allenai/specter_plus_plus")
30
  adapter_name = model.load_adapter("allenai/spp_proximity", source="hf", set_active=True)
31
  ```
32
+ ## SPECTER 2.0
33
 
34
+ <!-- Provide a quick summary of what the model is/does. -->
35
 
36
+ SPECTER 2.0 is the successor to [SPECTER](allenai/specter) and is capable of generating task specific embeddings for scientific tasks when paired with [adapters](https://huggingface.co/models?search=allenai/spp).
37
+ Given the combination of title and abstract of a scientific paper or a short texual query, the model can be used to generate effective embeddings to be used in downstream applications.
38
 
39
+ # Model Details
40
 
41
+ ## Model Description
42
+
43
+ SPECTER 2.0 has been trained on over 6M triplets of scientific paper citations, which are available [here](https://huggingface.co/datasets/allenai/scirepeval/viewer/cite_prediction_new/evaluation).
44
+ Post that it is trained on all the [SciRepEval](https://huggingface.co/datasets/allenai/scirepeval) training tasks, with task format specific adapters.
45
+
46
+ Task Formats trained on:
47
+ - Classification
48
+ - Regression
49
+ - Proximity
50
+ - Adhoc Search
51
+
52
+ This is a retrieval specific adapter. For tasks where given a paper query, other relevant papers have to be retrieved from a corpus, use this adapter to generate the embeddings.
53
+
54
+
55
+ It builds on the work done in [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://api.semanticscholar.org/CorpusID:254018137) and we evaluate the trained model on this benchmark as well.
56
+
57
+
58
+
59
+ - **Developed by:** Amanpreet Singh, Mike D'Arcy, Arman Cohan, Doug Downey, Sergey Feldman
60
+ - **Shared by :** Allen AI
61
+ - **Model type:** bert-base-uncased + adapters
62
+ - **License:** Apache 2.0
63
+ - **Finetuned from model [optional]:** [allenai/scibert](https://huggingface.co/allenai/scibert_scivocab_uncased).
64
+
65
+ ## Model Sources [optional]
66
+
67
+ <!-- Provide the basic links for the model. -->
68
+
69
+ - **Repository:** [https://github.com/allenai/SPECTER2_0] (https://github.com/allenai/SPECTER2_0)
70
+ - **Paper [optional]:** [https://api.semanticscholar.org/CorpusID:254018137](https://api.semanticscholar.org/CorpusID:254018137)
71
+ - **Demo [optional]:** [Usage] (https://github.com/allenai/SPECTER2_0/blob/main/README.md)
72
+
73
+ # Uses
74
+
75
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
76
+
77
+ ## Direct Use
78
+
79
+ |Model|Type|Name and HF link|
80
+ |--|--|--|
81
+ |Base|Transformer|[allenai/specter_plus_plus](https://huggingface.co/allenai/specter_plus_plus)|
82
+ |Classification|Adapter|[allenai/spp_classification](https://huggingface.co/allenai/spp_classification)|
83
+ |Regression|Adapter|[allenai/spp_regression](https://huggingface.co/allenai/spp_regression)|
84
+ |Retrieval|Adapter|[allenai/spp_proximity](https://huggingface.co/allenai/spp_proximity)|
85
+ |Adhoc Query|Adapter|[allenai/spp_adhoc_query](https://huggingface.co/allenai/spp_adhoc_query)|
86
+
87
+ ```python
88
+ from transformers import AutoTokenizer, AutoModel
89
+
90
+ # load model and tokenizer
91
+ tokenizer = AutoTokenizer.from_pretrained('allenai/specter_plus_plus')
92
+
93
+ #load base model
94
+ model = AutoModel.from_pretrained('allenai/specter_plus_plus')
95
+
96
+ #load the adapter(s) as per the required task, provide an identifier for the adapter in load_as argument and activate it
97
+ model.load_adapter("allenai/spp_proximity", source="hf", load_as="spp_proximity", set_active=True)
98
+
99
+ papers = [{'title': 'BERT', 'abstract': 'We introduce a new language representation model called BERT'},
100
+ {'title': 'Attention is all you need', 'abstract': ' The dominant sequence transduction models are based on complex recurrent or convolutional neural networks'}]
101
+
102
+ # concatenate title and abstract
103
+ text_batch = [d['title'] + tokenizer.sep_token + (d.get('abstract') or '') for d in papers]
104
+ # preprocess the input
105
+ inputs = self.tokenizer(text_batch, padding=True, truncation=True,
106
+ return_tensors="pt", return_token_type_ids=False, max_length=512)
107
+ output = model(**inputs)
108
+ # take the first token in the batch as the embedding
109
+ embeddings = output.last_hidden_state[:, 0, :]
110
+ ```
111
+
112
+ ## Downstream Use [optional]
113
+
114
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
115
+
116
+ For evaluation and downstream usage, please refer to [https://github.com/allenai/scirepeval/blob/main/evaluation/INFERENCE.md](https://github.com/allenai/scirepeval/blob/main/evaluation/INFERENCE.md).
117
+
118
+ # Training Details
119
+
120
+ ## Training Data
121
+
122
+ <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
123
+
124
+ The base model is trained on citation links between papers and the adapters are trained on 8 large scale tasks across the four formats.
125
+ All the data is a part of SciRepEval benchmark and is available [here](https://huggingface.co/datasets/allenai/scirepeval).
126
+
127
+ The citation link are triplets in the form
128
+
129
+ ```json
130
+ {"query": {"title": ..., "abstract": ...}, "pos": {"title": ..., "abstract": ...}, "neg": {"title": ..., "abstract": ...}}
131
+ ```
132
+
133
+ consisting of a query paper, a positive citation and a negative which can be from the same/different field of study as the query or citation of a citation.
134
+
135
+ ## Training Procedure
136
+
137
+ Please refer to the [SPECTER paper](https://api.semanticscholar.org/CorpusID:215768677).
138
+
139
+
140
+ ### Training Hyperparameters
141
+
142
+
143
+ The model is trained in two stages using [SciRepEval](https://github.com/allenai/scirepeval/blob/main/training/TRAINING.md):
144
+ - Base Model: First a base model is trained on the above citation triplets.
145
+ ``` batch size = 1024, max input length = 512, learning rate = 2e-5, epochs = 2 warmup steps = 10% fp16```
146
+ - Adapters: Thereafter, task format specific adapters are trained on the SciRepEval training tasks, where 600K triplets are sampled from above and added to the training data as well.
147
+ ``` batch size = 256, max input length = 512, learning rate = 1e-4, epochs = 6 warmup = 1000 steps fp16```
148
+
149
+
150
+ # Evaluation
151
+
152
+ We evaluate the model on [SciRepEval](https://github.com/allenai/scirepeval), a large scale eval benchmark for scientific embedding tasks which which has [SciDocs] as a subset.
153
+ We also evaluate and establish a new SoTA on [MDCR](https://github.com/zoranmedic/mdcr), a large scale citation recommendation benchmark.
154
+
155
+ |Model|SciRepEval In-Train|SciRepEval Out-of-Train|SciRepEval Avg|MDCR(MAP, Recall@5)|
156
+ |--|--|--|--|--|
157
+ |[BM-25](https://api.semanticscholar.org/CorpusID:252199740)|n/a|n/a|n/a|(33.7, 28.5)|
158
+ |[SPECTER](https://huggingface.co/allenai/specter)|54.7|57.4|68.0|(30.6, 25.5)|
159
+ |[SciNCL](https://huggingface.co/malteos/scincl)|55.6|57.8|69.0|(32.6, 27.3)|
160
+ |[SciRepEval-Adapters](https://huggingface.co/models?search=scirepeval)|61.9|59.0|70.9|(35.3, 29.6)|
161
+ |[SPECTER 2.0-base](https://huggingface.co/allenai/specter_plus_plus)|56.3|58.0|69.2|(38.0, 32.4)|
162
+ |[SPECTER 2.0-Adapters](https://huggingface.co/models?search=allen/spp)|**62.3**|**59.2**|**71.2**|**(38.4, 33.0)**|
163
+
164
+ Please cite the following works if you end up using SPECTER 2.0:
165
+
166
+ [SPECTER paper](https://api.semanticscholar.org/CorpusID:215768677):
167
+
168
+ ```bibtex
169
+ @inproceedings{specter2020cohan,
170
+ title={{SPECTER: Document-level Representation Learning using Citation-informed Transformers}},
171
+ author={Arman Cohan and Sergey Feldman and Iz Beltagy and Doug Downey and Daniel S. Weld},
172
+ booktitle={ACL},
173
+ year={2020}
174
+ }
175
+ ```
176
+ [SciRepEval paper](https://api.semanticscholar.org/CorpusID:254018137)
177
+ ```bibtex
178
+ @article{Singh2022SciRepEvalAM,
179
+ title={SciRepEval: A Multi-Format Benchmark for Scientific Document Representations},
180
+ author={Amanpreet Singh and Mike D'Arcy and Arman Cohan and Doug Downey and Sergey Feldman},
181
+ journal={ArXiv},
182
+ year={2022},
183
+ volume={abs/2211.13308}
184
+ }
185
+ ```
186
 
 
187