|
--- |
|
language: |
|
- zh |
|
tags: |
|
- LinkTransformer |
|
- Office Title Disambiguation/Similarity |
|
- 古代官职 |
|
- 古文 |
|
- 文言文 |
|
- ancient |
|
- classical |
|
license: cc-by-nc-sa-4.0 |
|
--- |
|
|
|
# <font color="IndianRed"> OfficeTitleDis (Classical Chinese Office Title Disambiguation/Similarity)</font> |
|
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1ql7NkLOGdEf2IaPg_9khGxev3OkZIaXu?usp=sharing) |
|
|
|
This model has been fine-tuned using methodologies from the paper ["LinkTransformer: A Unified Package for Record Linkage with Transformer Language Models"](https://scholar.harvard.edu/sites/scholar.harvard.edu/files/dell/files/linkt.pdf) by Abhishek Arora and Melissa Dell from Harvard University. |
|
|
|
### <font color="IndianRed">Model Description </font> |
|
This model is designed to find the top \(N\) most similar Classical Chinese office titles in a given data frame. Given an input DataFrame containing \(K\) office titles, the model outputs the top \(N\) most similar office titles in the input DataFrame for every office title. |
|
|
|
### <font color="IndianRed">Fine-tuning Data </font> |
|
The data used for fine-tuning this model is supported by the China Biographical Database (CBDB) at Harvard University. All office titles from the training data are from the periods of the Song, Ming, and Qing dynasties. |
|
|
|
--- |
|
|
|
### <font color="IndianRed">Usage</font> |
|
|
|
The following section demonstrates how to directly load the OfficeTitleDis model. |
|
|
|
Please ensure that you have the necessary libraries installed and model downloaded in your Python environment. If not, you can install it using pip: |
|
|
|
```python |
|
git lfs install |
|
git clone https://huggingface.co/cbdb/OfficeTitleDis |
|
pip install linktransformer |
|
pip install hanziconv |
|
``` |
|
|
|
Now, let's load our model and make some predictions: |
|
|
|
```python |
|
# Import necessary libraries from linktransformer |
|
import linktransformer as lt |
|
|
|
# predict |
|
df_lm_matched = lt.merge(df1, df2, merge_type='1:m', on="office_name", model="/content/OfficeTitleDis/model", left_on=None, right_on=None) |
|
display(df_lm_matched.head()) |
|
``` |
|
--- |
|
|
|
|
|
### <font color="IndianRed">Authors </font> |
|
Queenie Luo (queenieluo[at]g.harvard.edu) |
|
<br> |
|
Hongsu Wang |
|
<br> |
|
Peter Bol |
|
<br> |
|
CBDB Group |
|
|
|
### <font color="IndianRed">License </font> |
|
Copyright (c) 2023 CBDB |
|
|
|
Except where otherwise noted, content on this repository is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0). |
|
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/ or |
|
send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. |