KoichiYasuoka
/

deberta-base-japanese-juman-ud-goeswith

Token Classification Transformers PyTorch

Japanese deberta-v2 japanese wikipedia cc100 oscar pos dependency-parsing Inference Endpoints

Model card Files Files and versions Community

KoichiYasuoka commited on Feb 19, 2023

Commit

71f2fa2

•

1 Parent(s): b87dbd2

bug fix

Files changed (1) hide show

maker.py +1 -1

maker.py CHANGED Viewed

@@ -15,7 +15,7 @@ class UDgoeswithDataset(object):
         t=s.split("\t")
         if len(t)==10 and t[0].isdecimal():
           c.append(t)
-        elif c!=[]:
           v=tokenizer([t[1] for t in c],add_special_tokens=False)["input_ids"]
           for i in range(len(v)-1,-1,-1):
             for j in range(1,len(v[i])):

         t=s.split("\t")
         if len(t)==10 and t[0].isdecimal():
           c.append(t)
+        elif c!=[] and s.strip()=="":
           v=tokenizer([t[1] for t in c],add_special_tokens=False)["input_ids"]
           for i in range(len(v)-1,-1,-1):
             for j in range(1,len(v[i])):