古典中国語(漢文)の係り受け解析における正方行列アルゴリズム

[『青空文庫ModernBERTモデルによる国語研長単位係り受け解析』](https://ipsj.ixsq.nii.ac.jp/records/2000216)の正方行列アルゴリズムを、古典中国語(漢文)ModernBERTに適用して、係り受け解析モデルを試作してみた。ModernBERTだと90×90の正方行列が乗るので、どのくらいの解析精度になるのか試してみたかったのだ。Google Colaboratoryだと、こんな感じ。

```
!pip install transformers triton
models=[
  "KoichiYasuoka/modernbert-small-classical-chinese-ud-square",
  "KoichiYasuoka/modernbert-base-classical-chinese-ud-square",
  "KoichiYasuoka/modernbert-large-classical-chinese-ud-square"
]
import os,sys,subprocess
url="https://github.com/UniversalDependencies/UD_Classical_Chinese-Kyoto"
d=os.path.basename(url)
!test -d {d} || git clone --depth=1 {url}
!for F in train dev test ; do cp {d}/*-$$F.conllu $$F.conllu ; done
url="https://universaldependencies.org/conll18/conll18_ud_eval.py"
c=os.path.basename(url)
!test -f {c} || curl -LO {url}
with open("test.conllu","r",encoding="utf-8") as r:
  s=[t[8:].strip() for t in r if t.startswith("# text =")]
for mdl in models:
  from transformers import pipeline
  nlp=pipeline("universal-dependencies",mdl,trust_remote_code=True)
  for f in ["dev.conllu","test.conllu"]:
    with open(f,"r",encoding="utf-8") as r:
      s=[t[8:].strip() for t in r if t.startswith("# text =")]
    with open("result-"+f,"w",encoding="utf-8") as w:
      for t in s:
        try:
          w.write(nlp(t,aggregation_strategy="simple"))
        except:
          break
  os.system(f"mkdir -p result/{mdl}")
  with open(f"result/{mdl}/result.txt","w",encoding="utf-8") as w:
    for f in ["dev.conllu","test.conllu"]:
      p=subprocess.run([sys.executable,c,"-v",f,"result-"+f],
        encoding="utf-8",stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
      print(f"\n*** {mdl} ({f})",p.stdout,sep="\n",file=w)
!( cd result && cat `find {" ".join(models)} -name result.txt` )
```

[lzh_kyoto-ud-dev.conllu](https://github.com/UniversalDependencies/UD_Classical_Chinese-Kyoto/blob/master/lzh_kyoto-ud-dev.conllu)と[lzh_kyoto-ud-test.conllu](https://github.com/UniversalDependencies/UD_Classical_Chinese-Kyoto/blob/master/lzh_kyoto-ud-test.conllu)の両方で評価したところ、私(安岡孝一)の手元では以下の結果が出力された。

```
*** KoichiYasuoka/modernbert-small-classical-chinese-ud-square (dev.conllu)
Metric     | Precision |    Recall |  F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens     |     96.19 |     96.75 |     96.47 |
Sentences  |    100.00 |    100.00 |    100.00 |
Words      |     96.19 |     96.75 |     96.47 |
UPOS       |     89.25 |     89.77 |     89.51 |     92.79
XPOS       |      0.00 |      0.00 |      0.00 |      0.00
UFeats     |     90.35 |     90.87 |     90.61 |     93.92
AllTags    |      0.00 |      0.00 |      0.00 |      0.00
Lemmas     |     94.49 |     95.04 |     94.76 |     98.23
UAS        |     78.39 |     78.85 |     78.62 |     81.49
LAS        |     73.17 |     73.59 |     73.38 |     76.07
CLAS       |     72.45 |     72.85 |     72.65 |     75.53
MLAS       |     69.77 |     70.16 |     69.96 |     72.74
BLEX       |     71.49 |     71.89 |     71.69 |     74.54

*** KoichiYasuoka/modernbert-small-classical-chinese-ud-square (test.conllu)
Metric     | Precision |    Recall |  F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens     |     97.85 |     98.01 |     97.93 |
Sentences  |    100.00 |    100.00 |    100.00 |
Words      |     97.85 |     98.01 |     97.93 |
UPOS       |     91.12 |     91.27 |     91.19 |     93.12
XPOS       |      0.00 |      0.00 |      0.00 |      0.00
UFeats     |     92.52 |     92.68 |     92.60 |     94.56
AllTags    |      0.00 |      0.00 |      0.00 |      0.00
Lemmas     |     96.12 |     96.28 |     96.20 |     98.23
UAS        |     81.65 |     81.78 |     81.71 |     83.44
LAS        |     76.38 |     76.50 |     76.44 |     78.06
CLAS       |     75.73 |     75.60 |     75.67 |     77.31
MLAS       |     73.28 |     73.14 |     73.21 |     74.80
BLEX       |     74.75 |     74.62 |     74.68 |     76.31

*** KoichiYasuoka/modernbert-base-classical-chinese-ud-square (dev.conllu)
Metric     | Precision |    Recall |  F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens     |     96.22 |     97.08 |     96.65 |
Sentences  |    100.00 |    100.00 |    100.00 |
Words      |     96.22 |     97.08 |     96.65 |
UPOS       |     89.25 |     90.05 |     89.65 |     92.75
XPOS       |      0.00 |      0.00 |      0.00 |      0.00
UFeats     |     90.50 |     91.31 |     90.90 |     94.05
AllTags    |      0.00 |      0.00 |      0.00 |      0.00
Lemmas     |     94.51 |     95.36 |     94.93 |     98.22
UAS        |     79.04 |     79.75 |     79.39 |     82.14
LAS        |     73.93 |     74.59 |     74.26 |     76.83
CLAS       |     73.14 |     73.80 |     73.47 |     76.23
MLAS       |     70.66 |     71.29 |     70.97 |     73.65
BLEX       |     72.16 |     72.80 |     72.48 |     75.21

*** KoichiYasuoka/modernbert-base-classical-chinese-ud-square (test.conllu)
Metric     | Precision |    Recall |  F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens     |     98.17 |     98.42 |     98.30 |
Sentences  |    100.00 |    100.00 |    100.00 |
Words      |     98.17 |     98.42 |     98.30 |
UPOS       |     91.62 |     91.86 |     91.74 |     93.33
XPOS       |      0.00 |      0.00 |      0.00 |      0.00
UFeats     |     92.80 |     93.04 |     92.92 |     94.53
AllTags    |      0.00 |      0.00 |      0.00 |      0.00
Lemmas     |     96.44 |     96.69 |     96.56 |     98.24
UAS        |     82.55 |     82.76 |     82.65 |     84.09
LAS        |     77.37 |     77.57 |     77.47 |     78.82
CLAS       |     76.85 |     76.74 |     76.79 |     78.11
MLAS       |     74.36 |     74.25 |     74.30 |     75.57
BLEX       |     75.83 |     75.72 |     75.78 |     77.08

*** KoichiYasuoka/modernbert-large-classical-chinese-ud-square (dev.conllu)
Metric     | Precision |    Recall |  F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens     |     96.13 |     96.79 |     96.46 |
Sentences  |    100.00 |    100.00 |    100.00 |
Words      |     96.13 |     96.79 |     96.46 |
UPOS       |     89.13 |     89.75 |     89.44 |     92.72
XPOS       |      0.00 |      0.00 |      0.00 |      0.00
UFeats     |     90.34 |     90.96 |     90.65 |     93.98
AllTags    |      0.00 |      0.00 |      0.00 |      0.00
Lemmas     |     94.42 |     95.07 |     94.74 |     98.22
UAS        |     78.72 |     79.26 |     78.99 |     81.89
LAS        |     73.49 |     73.99 |     73.74 |     76.45
CLAS       |     72.66 |     73.15 |     72.90 |     75.81
MLAS       |     70.18 |     70.65 |     70.42 |     73.22
BLEX       |     71.66 |     72.14 |     71.90 |     74.76

*** KoichiYasuoka/modernbert-large-classical-chinese-ud-square (test.conllu)
Metric     | Precision |    Recall |  F1 Score | AligndAcc
-----------+-----------+-----------+-----------+-----------
Tokens     |     97.92 |     98.19 |     98.05 |
Sentences  |    100.00 |    100.00 |    100.00 |
Words      |     97.92 |     98.19 |     98.05 |
UPOS       |     91.30 |     91.55 |     91.43 |     93.24
XPOS       |      0.00 |      0.00 |      0.00 |      0.00
UFeats     |     92.41 |     92.66 |     92.54 |     94.37
AllTags    |      0.00 |      0.00 |      0.00 |      0.00
Lemmas     |     96.18 |     96.45 |     96.32 |     98.23
UAS        |     82.00 |     82.22 |     82.11 |     83.74
LAS        |     76.90 |     77.12 |     77.01 |     78.54
CLAS       |     76.28 |     76.25 |     76.26 |     77.81
MLAS       |     73.78 |     73.75 |     73.77 |     75.26
BLEX       |     75.28 |     75.25 |     75.26 |     76.79
```

LAS/MLAS/BLEXを表にしてみよう。

| |パラメータ数|トークン幅|lzh_kyoto-ud-dev.conllu|lzh_kyoto-ud-test.conllu|
|:-:|:-:|:-:|:-:|:-:|
|[modernbert-small-classical-chinese-ud-square](https://huggingface.co/KoichiYasuoka/modernbert-small-classical-chinese-ud-square)|1913万|8192|73.38/69.96/71.69|76.44/73.21/74.68|
|[modernbert-base-classical-chinese-ud-square](https://huggingface.co/KoichiYasuoka/modernbert-base-classical-chinese-ud-square)|1.24億|8192|74.26/70.97/72.48|77.47/74.30/75.78|
|[modernbert-large-classical-chinese-ud-square](https://huggingface.co/KoichiYasuoka/modernbert-large-classical-chinese-ud-square)|3.52億|8192|73.74/70.42/71.90|77.01/73.77/75.26|

[6月9日の記事](https://qiita.com/KoichiYasuoka/items/0c3e61d27096b7b4adfc)と比較すると、modernbert-baseやmodernbert-largeに関しては、上三角行列より正方行列の方が精度が高いようだ。でも、modernbert-smallでは逆である。うーん、どうなってるんだろ。