Package: depccg
depccg v2Codebase forA* CCG Parsing with a Supertag and Dependency Factored Model2021/07/12 Updates (v2)Increased stability and efficiency(Replaced OpenMP with multiprocessing)More integration with AllenNLPThe parser is now callable from within apredictor(seehere)More friendly way to define your own grammar (wrt. languages or treebanks)Seedepccg/grammar/{en,ja}.pyfor example grammars.RequirementsPython >= 3.6.0A C++ compiler supportingC++11 standard(in case of gcc, must be >= 4.8)InstallationUsing pip:➜pipinstallcythonnumpydepccgUsageUsing a pretrained English parserCurrently following models are available for English:NameDescriptionunlabeled/labeled F1 on CCGbankDownloadbasicmodel trained on the combination of CCGbank and tri-training dataset (Yoshikawa et al., 2017)94.0%/88.8%link(189M)elmobasic model with its embeddings replaced with ELMo (Peters et al., 2018)94.98%/90.51%link(649M)rebankbasic model trained on Rebanked CCGbank (Honnibal et al., 2010)-link(337M)elmo_rebankELMo model trained on Rebanked CCGbank-link(1G)The basic model is available by:➜depccg_endownloadTo use:➜echo"this is a test sentence ."|depccg_enID=1,Prob=-0.0006299018859863281(<TS[dcl]02>(<TS[dcl]02>(<LNPXXXXthisNP>)(<TS[dcl]\NP02>(<L(S[dcl]\NP)/NPXXXXis(S[dcl]\NP)/NP>)(<TNP02>(<LNP[nb]/NXXXXaNP[nb]/N>)(<TN02>(<LN/NXXXXtestN/N>)(<LNXXXXsentenceN>)))))(<L.XXXX..>))You can download other models by specifying their names:➜depccg_endownloadelmoTo use, make sure to installallennlp:➜echo"this is a test sentence ."|depccg_en--modelelmoYou can also specify in the--modeloption the path of a model file (in tar.gz) that is available from links above.Using a GPU (by--gpuoption) is recommended if possible.There are several output formats (seebelow).➜echo"this is a test sentence ."|depccg_en--formatderivID=1,Prob=-0.0006299018859863281thisisatestsentence.NP(S[dcl]\NP)/NPNP[nb]/NN/NN.---------------->N-------------------------->NP------------------------------------------>S[dcl]\NP
------------------------------------------------<S[dcl]---------------------------------------------------<rp>S[dcl]By default, the input is expected to be pre-tokenized. If you want to process untokenized sentences, you can pass--tokenizeoption.The POS and NER tags in the output are filled withXXby default. You can replace them with ones predicted usingSpaCy:➜echo"this is a test sentence ."|depccg_en--annotatorspacyID=1,Prob=-0.0006299018859863281(<TS[dcl]02>(<TS[dcl]02>(<LNPDTDTthisNP>)(<TS[dcl]\NP02>(<L(S[dcl]\NP)/NPVBZVBZis(S[dcl]\NP)/NP>)(<TNP02>(<LNP[nb]/NDTDTaNP[nb]/N>)(<TN02>(<LN/NNNNNtestN/N>)(<LNNNNNsentenceN>)))))(<L.....>))The parser uses a SpaCy'sen_core_web_smmodel.Orelse, you can use POS/NER taggers implemented inC&C, which may be useful in some sorts of parsing experiments:➜exportCANDC=/path/to/candc
➜echo"this is a test sentence ."|depccg_en--annotatorcandcID=1,logprob=-0.0006299018859863281(<TS[dcl]02>(<TS[dcl]02>(<LNPDTDTthisNP>)(<TS[dcl]\NP02>(<L(S[dcl]\NP)/NPVBZVBZis(S[dcl]\NP)/NP>)(<TNP02>(<LNP[nb]/NDTDTaNP[nb]/N>)(<TN02>(<LN/NNNNNtestN/N>)(<LNNNNNsentenceN>)))))(<L.....>))By default, depccg expects the POS and NER models are placed in$CANDC/models/posand$CANDC/models/ner, but you can explicitly specify them by settingCANDC_MODEL_POSandCANDC_MODEL_NERenvironmental variables.It is also possible to obtain logical formulas usingccg2lambda's semantic parsing algorithm.➜echo"This is a test sentence ."|depccg_en--formatccg2lambda--annotatorspacyID=0logprobability=-0.0006299018859863281
existsx.(_this(x)&existsz1.(_sentence(z1)&_test(z1)&(x=z1)))Using a pretrained Japanese parserThe best performing model is available by:➜depccg_jadownloadIt can be downloaded directlyhere(56M).The parser provides the almost same interface as with the English one, with slight differences including the default output format, which is now one compatible with the Japanese CCGbank:➜echo"これはテストの文です。"|depccg_jaID=1,Prob=-53.98793411254883{<S[mod=nm,form=base,fin=t]{<S[mod=nm,form=base,fin=f]{<NP[case=nc,mod=nm,fin=f]{NP[case=nc,mod=nm,fin=f]これ/これ/**}{NP[case=nc,mod=nm,fin=f]\NP[case=nc,mod=nm,fin=f]は/は/**}}{<S[mod=nm,form=base,fin=f]\NP[case=nc,mod=nm,fin=f]{<NP[case=nc,mod=nm,fin=f]{<NP[case=nc,mod=nm,fin=f]{NP[case=nc,mod=nm,fin=f]テスト/テスト/**}{NP[case=nc,mod=nm,fin=f]\NP[case=nc,mod=nm,fin=f]の/の/**}}{NP[case=nc,mod=nm,fin=f]\NP[case=nc,mod=nm,fin=f]文/文/**}}{(S[mod=nm,form=base,fin=f]\NP[case=nc,mod=nm,fin=f])\NP[case=nc,mod=nm,fin=f]です/です/**}}}{S[mod=nm,form=base,fin=t]\S[mod=nm,form=base,fin=f]。/。/**}}You can pass pre-tokenized sentences as well:➜echo"これ は テスト の 文 です 。"|depccg_ja--pre-tokenizedID=1,Prob=-53.98793411254883{<S[mod=nm,form=base,fin=t]{<S[mod=nm,form=base,fin=f]{<NP[case=nc,mod=nm,fin=f]{NP[case=nc,mod=nm,fin=f]これ/これ/**}{NP[case=nc,mod=nm,fin=f]\NP[case=nc,mod=nm,fin=f]は/は/**}}{<S[mod=nm,form=base,fin=f]\NP[case=nc,mod=nm,fin=f]{<NP[case=nc,mod=nm,fin=f]{<NP[case=nc,mod=nm,fin=f]{NP[case=nc,mod=nm,fin=f]テスト/テスト/**}{NP[case=nc,mod=nm,fin=f]\NP[case=nc,mod=nm,fin=f]の/の/**}}{NP[case=nc,mod=nm,fin=f]\NP[case=nc,mod=nm,fin=f]文/文/**}}{(S[mod=nm,form=base,fin=f]\NP[case=nc,mod=nm,fin=f])\NP[case=nc,mod=nm,fin=f]です/です/**}}}{S[mod=nm,form=base,fin=t]\S[mod=nm,form=base,fin=f]。/。/**}}Available output formatsauto- the most standard format following AUTO format in the English CCGbankauto_extended- extension of auto format with combinator info and POS/NER tagsderiv- visualized derivations in ASCII artxml- XML format compatible with C&C's XML format (only for English parsing)conll- CoNLL formathtml- visualized trees in MathMLprolog- Prolog-like formatjigg_xml- XML format compatible withJiggptb- Penn Treebank-style formatccg2lambda- logical formula converted from a derivation usingccg2lambdajigg_xml_ccg2lambda- jigg_xml format with ccg2lambda logical formula insertedjson- JSON formatja- a format adopted in Japanese CCGbank (only for Japanese)Programmatic UsagePlease look intodepccg/__main__.py.Train your own parsing modelYou can use myallennlp-based supertagger and extend it.To train a supertagger, preparethe English CCGbankand downloadvocab:➜catccgbank/data/AUTO/{0[2-9],1[0-9],20,21}/*>wsj_02-21.auto
➜catccgbank/data/AUTO/00/*>wsj_00.auto➜wgethttp://cl.naist.jp/~masashi-y/resources/depccg/vocabulary.tar.gz
➜tarxvfvocabulary.tar.gzthen,➜vocab=vocabularytrain_data=wsj_02-21.autotest_data=wsj_00.autogpu=0\encoder_type=lstmtoken_embedding_type=char\allennlptrain--include-packagedepccg--serialization-dirresultsdepccg/allennlp/configs/supertagger.jsonnetThe training configs are passed either through environmental variables or directly writing to jsonnet config files, which are available insupertagger.jsonnetorsupertagger_tritrain.jsonnet.
The latter is a config file for usingtri-training silver data(309M) constructed in (Yoshikawa et al., 2017), on top of the English CCGbank.To use the trained supertagger,➜echo'{"sentence": "this is a test sentence ."}'>input.jsonl
➜allennlppredictresults/model.tar.gz--include-packagedepccg--output-fileweights.jsoninput.jsonlor alternatively, you can perform CCG parsing:➜allennlppredict--include-packagedepccg--predictorparser-predictor--predictor-args'{"grammar_json_path": "depccg/models/config_en.jsonnet"}'model.tar.gzinput.jsonlEvaluation in terms of predicate-argument dependenciesThe standard CCG parsing evaluation can be performed with the following script:➜catccgbank/data/PARG/00/*>wsj_00.parg
➜exportCANDC=/path/to/candc
➜python-mdepccg.tools.evaluatewsj_00.pargwsj_00.predicted.autoThe script is dependent onC&C'sgenerateprogram, which is only available by compiling the C&C program from the source.(Currently, the above page is down. You can find the C&C parserhereorhere)MiscellaneousDiff toolIn error analysis, you must want to see diffs between trees in an intuitive way.depccg.tools.diffdoes exactly this:➜python-mdepccg.tools.difffile1.autofile2.auto>diff.htmlwhich outputs:where trees in the same lines of the files are compared and the diffs are marked in color.CitationIf you make use of this software, please cite the following:@inproceedings{yoshikawa:2017acl,
      author={Yoshikawa, Masashi and Noji, Hiroshi and Matsumoto, Yuji},
      title={A* CCG Parsing with a Supertag and Dependency Factored Model},
      booktitle={Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
      publisher={Association for Computational Linguistics},
      year={2017},
      pages={277--287},
      location={Vancouver, Canada},
      doi={10.18653/v1/P17-1026},
      url={http://aclweb.org/anthology/P17-1026}
    }LicenceMIT LicenceContactFor questions and usage issues, please contactyoshikawa@tohoku.jp.AcknowledgementIn creating the parser, I owe very much to:EasyCCG: from which I learned everythingNLTK: for nice pretty printing for parse derivation