Package: cwb-ccc
Collocation and Concordance Computationcwb-cccis a Python 3 wrapper around theIMS Open Corpus Workbench (CWB).  Main purpose of the module is to run queries (including queries with more than two anchor points), extract concordance lines, and score frequency lists (particularly to extract collocates and keywords).TheQuickstartshould get you started. For a more detailed overview of the functionality, see theVignette.InstallationQuickstartAccessing CorporaQueries and SubCorporaConcordancingCollocatesKeywordsTestingAcknowledgementsInstallationThe module needs a working installation of theCWBand operates on CWB-indexed corpora. If you want to run queries with more than two anchor points, you will need CWB version 3.4.16 or later.You can install cwb-ccc with pip fromPyPI:python -m pip install cwb-cccYou can also clone the source fromgithub,cdin the respective folder, and build your own wheel:python -m pip install pipenv
pipenv install --dev
pipenv run python3 -m cython -2 ccc/cl.pyx
pipenv run python3 setup.py bdist_wheelQuickstartAccessing CorporaTo list all available corpora, you can usefromcccimportCorporacorpora=Corpora(registry_dir="/usr/local/share/cwb/registry/")All further methods rely on theCorpusclass, which establishes the connection to your CWB-indexed corpus:fromcccimportCorpuscorpus=Corpus(corpus_name="GERMAPARL1386",registry_dir="/usr/local/share/cwb/registry/")This will raise aKeyErrorif the named corpus is not in the specified registry.Queries and SubCorporaThe usual starting point for using this module is to run a query withcorpus.query(), which accepts valid CQP queries such assubcorpus=corpus.query('[lemma="Arbeit"]',context_break='s')The result is aSubCorpus; at its core this is a pandas DataFrame with corpus positions (similar to CWB dumps of NQRs).ConcordancingYou can access concordance lines via theconcordance()method of the subcorpus.  This method returns a DataFrame with information about the query matches in context:subcorpus.concordance()matchmatchendword151151Er brachte diese Erfahrung in seine Arbeit im Ausschuß für Familie , Senioren , Frauen und Jugend sowie im Petitionsausschuß ein , wo er sich vor allem227227Seine Arbeit und sein Rat werden uns fehlen .14931493Ausschuß für Arbeit und Sozialordnung15551555Ausschuß für Arbeit und Sozialordnung15981598Ausschuß für Arbeit und Sozialordnung.........By default, this retrieves concordance lines in simple format in the order in which they appear in the corpus. A better approach issubcorpus.concordance(form='kwic', order='random')matchmatchendleft_wordnode_wordright_word8176981769Ich unterstütze daher nachträglich die Forderung , daß die Durchführung des Gesetzes auch künftig durch die Bundesanstalt fürArbeitvorgenommen wird ; denn beim Bund gibt es die entsprechend ausgebildeten Sachbearbeiter .87748774Glauben Sie im Ernst , Sie könnten am Ende ein Bündnis fürArbeit, eine Wende in der deutschen Politik , die Bekämpfung der Arbeitslosigkeit erreichen , wenn Sie nicht die Länder ,89948994alle Entscheidungen gemeinsam zu treffen , die sich gegen Schwarzarbeit und illegaleArbeitwenden , und gemeinsam nach einem Weg zu suchen ,8009880098: Was der Vermittlungsausschuß mit Mehrheit zum Meister-BAföG beschlossen hat , heißt , daß die bewährten Institutionen der Bundesanstalt fürArbeit, die die Ausbildungsförderung für Meister bis zum Jahr 1993 durchgeführt haben , die darin große Erfahrung haben , die6105661056Selbst wenn Sie ein Konstrukt anbieten , das tendenziell die zusätzliche Belastung der Bundesanstalt fürArbeitetwas geringer hielte als die Entlastung bei der gesetzlichen Rentenversicherung , so wäre dies bei einem deutlichen Aufwuchs der Arbeitslosigkeit...............Collocation AnalysesAfter executing a query, you can usesubcorpus.collocates()to extract collocates (see the vignette for parameter settings). The result is aDataFramewith lemmata as index and frequency signatures and association measures as columns:subcorpus.collocates()itemO11O12O21O22R1R2C1C2NE11E12E21E22z_scoret_scorelog_likelihoodsimple_llmin_sensitivityliddelldicelog_ratioconservative_log_ratiomutual_informationlocal_mutual_informationipmipm_referenceipm_expectedin_nodesmarginalfür467308311481027761489338771488321497094.54583771.454872.45414806119.44296.11208134.301130.0190.0524520.0475470.0556563.409252.263351.0051446.236659278.45579.695858.030877,437337827141106776148933787014183914970940.7933735.2077829.211411040.3455050.3365230.1245640.1172780.0054640.0002960.0099470.07641200.022880.98383655412.452553.852568.607870.337435626143307776148933565914405014970929.3328746.6675629.671433030.6771080.6383780.4610050.4404810.0058310.0006730.0102560.17089100.051161.6882942525.837775.43780005659und327442848146085776148933288014682914970914.9282761.0722865.071460684.418523.017915.145214.65550.0111110.0060440.0175051.1086600.33114410.596641237.119122.719237.302880in247522474146459776148933249814721114970912.9481763.0522485.051464483.071382.255967.728137.517220.0096080.0044990.0146610.89672400.2680056.4321230927.816611.516685.702498..........................................................................................The dataframe contains the counts and is annotated with all available association measures in thepandas-association-measurespackage (parameterams).Keyword AnalysesHaving created a subcorpussubcorpus=corpus.query(s_query='text_party',s_values={'CDU','CSU'})you can use itskeywords()method for retrieving keywords:subcorpus.keywords(order='conservative_log_ratio')itemO11O12O21O22R1R2C1C2NE11E12E21E22z_scoret_scorelog_likelihoodsimple_llmin_sensitivityliddelldicelog_ratioconservative_log_ratiomutual_informationlocal_mutual_informationipmipm_referenceipm_expecteddeswegen554129637108412413511084499214970814980025.395841325.666.60421083825.874523.9918341.530825.7940.001330.3219820.0026541.962930.4041660.33560118.4581330.08341.174614.152CSU2554109638010806941351108449635149165149800175.28641175.7459.7141079896.020874.9918746.654331.74250.0061670.1260680.0121470.815520.2123010.16279241.5126166.723503.954238.99CDU2604109139010805941351108449650149150149800179.42741171.6470.5731079786.015154.9969346.605531.72890.0062880.1244990.0123810.806060.2095110.16108641.88236287.643596.164339.12in867404841631106818413511084492498147302149800689.55140661.41808.451066416.757556.0264761.266342.18490.0209670.0722410.0395450.479370.1689010.09945286.225320966.815039.316675.6Wirtschaft394131225108424413511084496414973614980017.666641333.346.33341084035.075543.4160730.932819.10020.0009430.3334760.0018832.032570.1509820.3439113.4125943.145230.523427.236....................................................................................Just as with collocates, the result is aDataFramewith lemmata as index and frequency signatures and association measures as columns.TestingThe module ships with a small test corpus ("GERMAPARL1386"), which contains all speeches of the 86th session of the 13th German Bundestag on Feburary 8, 1996. This corpus consists of 149,800 tokens in 7332 paragraphs (s-attribute "p" with annotation "type" ("regular" or "interjection")) split into 11,364 sentences (s-attribute "s").  The p-attributes are "pos" and "lemma":corpus.available_attributes()typeattributeannotationactivep-AttwordFalseTruep-AttposFalseFalsep-AttlemmaFalseFalses-AttcorpusFalseFalses-Attcorpus_nameTrueFalses-AttsitzungFalseFalses-Attsitzung_dateTrueFalses-Attsitzung_periodTrueFalses-Attsitzung_sessionTrueFalses-AttdivFalseFalses-Attdiv_descTrueFalses-Attdiv_nTrueFalses-Attdiv_typeTrueFalses-Attdiv_whatTrueFalses-AtttextFalseFalses-Atttext_idTrueFalses-Atttext_nameTrueFalses-Atttext_parliamentary_groupTrueFalses-Atttext_partyTrueFalses-Atttext_positionTrueFalses-Atttext_roleTrueFalses-Atttext_whoTrueFalses-AttpFalseFalses-Attp_typeTrueFalses-AttsFalseFalseThe corpus is located in thisrepository.  All tests are written using this corpus (as well as some reference counts and scores from theUCS toolkitand some additional frequency lists).  Make sure you install all development dependencies (especiallypytest):python -m pip install pipenv
pipenv install --devYou can then simplymake build
make test
make coverageAcknowledgementsThe module includes a slight adaptation ofcwb-python, a Python port of Perl's CWB::CL; thanks toYannick Versleyfor the implementation.Special thanks toMarkus Opolkafor the original implementation ofassociation-measuresand for forcing me to write tests.The test corpus was extracted from theGermaParlcorpus (see thePolMine Project); many thanks toAndreas Blätte.This work was supported by theEmerging Fields Initiative (EFI)ofFriedrich-Alexander-Universität Erlangen-Nürnberg, project titleExploring theFukushima Effect(2017-2020).Further development of the package is funded by the Deutsche Forschungsgemeinschaft (DFG) within the projectReconstructing Arguments from Noisy Text, grant number 377333057 (2018-2024), as part of the Priority ProgramRobust Argumentation Machines(SPP-1999).