KoichiYasuoka
/

roberta-base-japanese-juman-ud-goeswith

Token Classification Transformers PyTorch

Japanese roberta japanese wikipedia cc100 pos dependency-parsing Inference Endpoints

Model card Files Files and versions Community

KoichiYasuoka commited on Feb 21, 2023

Commit

df69051

•

1 Parent(s): 0359194

常用漢字 support

Files changed (1) hide show

tokenizer.json +29 -1

tokenizer.json CHANGED Viewed

@@ -76,6 +76,34 @@
           "Regex": " {2,}"
         },
         "content": " "
       }
     ]
   },
@@ -128170,4 +128198,4 @@
       ]
     ]
   }
-}

           "Regex": " {2,}"
         },
         "content": " "
+      },
+      {
+        "type": "Replace",
+        "pattern": {
+          "String": "𠮟"
+        },
+        "content": "叱"
+      },
+      {
+        "type": "Replace",
+        "pattern": {
+          "String": "塡"
+        },
+        "content": "填"
+      },
+      {
+        "type": "Replace",
+        "pattern": {
+          "String": "剝"
+        },
+        "content": "剥"
+      },
+      {
+        "type": "Replace",
+        "pattern": {
+          "String": "頰"
+        },
+        "content": "頬"
       }
     ]
   },
       ]
     ]
   }
+}