File size: 3,717 Bytes
87a461c 3e88b44 87a461c 0366aed dc3b43a 0366aed 87a461c 51fd61b 87a461c 51fd61b 87a461c 51fd61b 87a461c 51fd61b 87a461c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 |
---
language:
- zh
tags:
- Seq2SeqLM
- 古文
- 文言文
- 中国古代官职地名拆分
- ancient
- classical
license: cc-by-nc-sa-4.0
---
# <font color="IndianRed"> OTAS (Office Title Address Splitter)</font>
[](https://colab.research.google.com/drive/1UoG3QebyBlK6diiYckiQv-5dRB9dA4iv?usp=sharing)
Our model <font color="cornflowerblue">OTAS (Office Title Address Splitter) </font> is a Named Entity Recognition Classical Chinese language model that is intended to <font color="IndianRed">split the address portion in Classical Chinese office titles.</font>. This model is first inherited from raynardj/classical-chinese-punctuation-guwen-biaodian Classical Chinese punctuation model, and finetuned using over a 25,000 high-quality punctuation pairs collected CBDB group (China Biographical Database).
### <font color="IndianRed"> Sample input txt file </font>
The sample input txt file can be downloaded here:
https://huggingface.co/cbdb/OfficeTitleAddressSplitter/blob/main/input.txt
### <font color="IndianRed"> How to use </font>
Here is how to use this model to get the features of a given text in PyTorch:
<font color="cornflowerblue"> 1. Import model and packages </font>
```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
PRETRAINED = "cbdb/OfficeTitleAddressSplitter"
tokenizer = AutoTokenizer.from_pretrained(PRETRAINED)
model = AutoModelForTokenClassification.from_pretrained(PRETRAINED)
```
<font color="cornflowerblue"> 2. Load Data </font>
```python
# Load your data here
test_list = ['漢軍鑲黃旗副都統', '兵部右侍郎', '盛京戶部侍郎']
```
<font color="cornflowerblue"> 3. Make a prediction </font>
```python
def predict_class(test):
tokens_test = tokenizer.encode_plus(
test,
add_special_tokens=True,
return_attention_mask=True,
padding=True,
max_length=128,
return_tensors='pt',
truncation=True
)
test_seq = torch.tensor(tokens_test['input_ids'])
test_mask = torch.tensor(tokens_test['attention_mask'])
inputs = {
"input_ids": test_seq,
"attention_mask": test_mask
}
with torch.no_grad():
# print(inputs.shape)
outputs = model(**inputs)
outputs = outputs.logits.detach().cpu().numpy()
softmax_score = softmax(outputs)
softmax_score = np.argmax(softmax_score, axis=2)[0]
return test_seq, softmax_score
for test_sen0 in test_list:
test_seq, pred_class_proba = predict_class(test_sen0)
test_sen = tokenizer.decode(test_seq[0]).split()
label = [idx2label[i] for i in pred_class_proba]
element_to_find = '。'
if element_to_find in label:
index = label.index(element_to_find)
test_sen_pred = [i for i in test_sen0]
test_sen_pred.insert(index, element_to_find)
test_sen_pred = ''.join(test_sen_pred)
else:
test_sen_pred = [i for i in test_sen0]
test_sen_pred = ''.join(test_sen_pred)
print(test_sen_pred)
```
漢軍鑲黃旗。副都統<br>
兵部右侍郎<br>
盛京。戶部侍郎<br>
### <font color="IndianRed">Authors </font>
Queenie Luo (queenieluo[at]g.harvard.edu)
<br>
Hongsu Wang
<br>
Peter Bol
<br>
CBDB Group
### <font color="IndianRed">License </font>
Copyright (c) 2023 CBDB
Except where otherwise noted, content on this repository is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/ or
send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. |