Package: cohesion-tools
Cohesion ToolsRequirementsPython: 3.8+Dependencies: Seepyproject.toml.Installationpipinstallcohesion-tools# or cohesion-tools[eval]UsageEvaluating Predicted DocumentsfrompathlibimportPathfromtypingimportListfromrhoknpimportDocumentfromrhoknp.cohesionimportExophoraReferentTypefromcohesion_tools.evaluatorsimportCohesionEvaluator,CohesionScoredocuments:List[Document]=[Document.from_knp(path.read_text())forpathinPath("your/dataset").glob("*.knp")]predicted_documents=your_model(documents)scorer=CohesionEvaluator(exophora_referent_types=[ExophoraReferentType(t)fortin("著者","読者","不特定:人","不特定:物")],pas_cases=["ガ","ヲ","ニ"],)score:CohesionScore=scorer.run(predicted_documents=predicted_documents,gold_documents=documents)score.to_dict()# Convert the evaluation result to a dictionaryscore.export_csv("score.csv")# Export the evaluation result to `score.csv`score.export_txt("score.txt")# Export the evaluation result to `score.txt`Extracting Labels From Base PhrasesfrompathlibimportPathfromtypingimportDict,ListfromrhoknpimportDocumentfromrhoknp.cohesionimportExophoraReferentType,Argumentfromcohesion_tools.extractorsimportPasExtractorpas_extractor=PasExtractor(cases=["ガ","ヲ","ニ"],exophora_referent_types=[ExophoraReferentType(t)fortin("著者","読者","不特定:人","不特定:物")],)examples=[]documents:List[Document]=[Document.from_knp(path.read_text())forpathinPath("your/dataset").glob("*.knp")]fordocumentindocuments:forbase_phraseindocument.base_phrases:ifpas_extractor.is_target(base_phrase)isTrue:rels:Dict[str,List[Argument]]=pas_extractor.extract_rels(base_phrase)examples.append(rels)your_trainer.train(your_model,examples)Reference@inproceedings{ueda-etal-2020-bert,title={{BERT}-based Cohesion Analysis of {J}apanese Texts},author={Ueda, Nobuhiro  andKawahara, Daisuke  andKurohashi, Sadao},booktitle={Proceedings of the 28th International Conference on Computational Linguistics},month=dec,year={2020},address={Barcelona, Spain (Online)},publisher={International Committee on Computational Linguistics},url={https://aclanthology.org/2020.coling-main.114},doi={10.18653/v1/2020.coling-main.114},pages={1323--1333},abstract={The meaning of natural language text is supported by cohesion among various kinds of entities, including coreference relations, predicate-argument structures, and bridging anaphora relations. However, predicate-argument structures for nominal predicates and bridging anaphora relations have not been studied well, and their analyses have been still very difficult. Recent advances in neural networks, in particular, self training-based language models including BERT (Devlin et al., 2019), have significantly improved many natural language processing tasks, making it possible to dive into the study on analysis of cohesion in the whole text. In this study, we tackle an integrated analysis of cohesion in Japanese texts. Our results significantly outperformed existing studies in each task, especially about 10 to 20 point improvement both for zero anaphora and coreference resolution. Furthermore, we also showed that coreference resolution is different in nature from the other tasks and should be treated specially.}}@inproceedings{ueda-etal-2023-kwja,title={{KWJA}: A Unified {J}apanese Analyzer Based on Foundation Models},author={Ueda, Nobuhiro  andOmura, Kazumasa  andKodama, Takashi  andKiyomaru, Hirokazu  andMurawaki, Yugo  andKawahara, Daisuke  andKurohashi, Sadao},booktitle={Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},month=jul,year={2023},address={Toronto, Canada},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2023.acl-demo.52},pages={538--548},abstract={We present KWJA, a high-performance unified Japanese text analyzer based on foundation models.KWJA supports a wide range of tasks, including typo correction, word segmentation, word normalization, morphological analysis, named entity recognition, linguistic feature tagging, dependency parsing, PAS analysis, bridging reference resolution, coreference resolution, and discourse relation analysis, making it the most versatile among existing Japanese text analyzers.KWJA solves these tasks in a multi-task manner but still achieves competitive or better performance compared to existing analyzers specialized for each task.KWJA is publicly available under the MIT license at https://github.com/ku-nlp/kwja.}}LicenseThis software is released under the MIT License, seeLICENSE.