Lucia Zheng
commited on
Commit
•
0b9e19e
1
Parent(s):
2e68954
Update model card
Browse files
README.md
CHANGED
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
### Legal-BERT
|
2 |
+
Model and tokenizer files for Legal-BERT model from [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset of 53,000+ Legal Holdings](https://arxiv.org/abs/2104.08671).
|
3 |
+
|
4 |
+
### Training Data
|
5 |
+
The pretraining corpus was constructed by ingesting the entire Harvard Law case corpus from 1965 to the present (https://case.law/). The size of this corpus (37GB) is substantial, representing 3,446,187 legal decisions across all federal and state courts, and is larger than the size of the BookCorpus/Wikipedia corpus originally used to train BERT (15GB).
|
6 |
+
|
7 |
+
### Training Objective
|
8 |
+
This model is initialized with [bert-base-uncased](https://huggingface.co/bert-base-uncased) and trained for an additional 1M steps on the MLM and NSP objective, with tokenization and sentence segmentation adapted for legal text (cf. the paper).
|
9 |
+
|
10 |
+
### Usage
|
11 |
+
Please see the [casehold repository](https://github.com/reglab/casehold) for scripts that support computing pretrain loss and finetuning on Legal-BERT for classification and multiple choice tasks described in the paper: Overruling, Terms of Service, CaseHOLD.
|
12 |
+
|
13 |
+
### Citation
|
14 |
+
@inproceedings{zhengguha2021,
|
15 |
+
title={When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset},
|
16 |
+
author={Lucia Zheng and Neel Guha and Brandon R. Anderson and Peter Henderson and Daniel E. Ho},
|
17 |
+
year={2021},
|
18 |
+
eprint={2104.08671},
|
19 |
+
archivePrefix={arXiv},
|
20 |
+
primaryClass={cs.CL},
|
21 |
+
booktitle={Proceedings of the 18th International Conference on Artificial Intelligence and Law},
|
22 |
+
publisher={Association for Computing Machinery},
|
23 |
+
note={(in press)}
|
24 |
+
}
|