KoichiYasuoka
/

deberta-base-japanese-juman-ud-goeswith

Token Classification Transformers PyTorch

Japanese deberta-v2 japanese wikipedia cc100 oscar pos dependency-parsing Inference Endpoints

Model card Files Files and versions Community

KoichiYasuoka commited on Feb 6, 2023

Commit

b87dbd2

•

1 Parent(s): 43e4326

常用漢字 support

Files changed (1) hide show

tokenizer.json +29 -1

tokenizer.json CHANGED Viewed

@@ -67,6 +67,34 @@
           "Regex": " {2,}"
         },
         "content": " "
       }
     ]
   },
@@ -128166,4 +128194,4 @@
       ]
     ]
   }
-}

           "Regex": " {2,}"
         },
         "content": " "
+      },
+      {
+        "type": "Replace",
+        "pattern": {
+          "String": "𠮟"
+        },
+        "content": "叱"
+      },
+      {
+        "type": "Replace",
+        "pattern": {
+          "String": "塡"
+        },
+        "content": "填"
+      },
+      {
+        "type": "Replace",
+        "pattern": {
+          "String": "剝"
+        },
+        "content": "剥"
+      },
+      {
+        "type": "Replace",
+        "pattern": {
+          "String": "頰"
+        },
+        "content": "頬"
       }
     ]
   },
       ]
     ]
   }
+}