Package: concordancer
ConcordancerThis library loads and indexes a corpus in RAM and provides concordance search to retrieve data from the corpus with (a subset of) Corpus Query Language (CQL).Installationpipinstall-UconcordancerUsageConcordanceris designed with this workflow in mind:The user is expected to preprocess the text data to match the corpus data required byconcordancer. Once this is done, subsequent tasks such as indexing the copus, writing query functions to search the corpus, and displaying results in an aligned keyword-in-context format are all done byconcordancer. The user could then further process the search results (exported as JSON byconcordancer) for other uses.Input corpus data structureconcordancerrequires the corpus to be structured (minimally) as:[# a corpus{# a text'text':[[<tk>,<tk>,<tk>,...],# a sentence in a text[<tk>,<tk>,<tk>,...],# another sentence in a text...[<tk>,<tk>,<tk>,...]# the last sentence in a text]},{...},# another text...]where<tk>is a dictionary representating a token, which may resemble something like:{'word':'hits','lemma':'hit','pos':'V'}This structure allows the corpus to be saved conveniently as anewline-delimited JSONfile (.jsonl), where each line of the file corresponds to a single text in the corpus, represented as a JSON object (i.e., a dictionary in Python). You can see an example of the corpus file saved in.jsonlhere. The code below uses a corpus saved in.jsonlformat for demonstration.Loading a corpus from fileThe code below uses anexample corpus, which is saved as anewline-delimited JSONfile (described in the previous section).importjsonfromconcordancer.demoimportdownload_demo_corpusfromconcordancer.concordancerimportConcordancerfromconcordancerimportserver# Load demo corpusfp=download_demo_corpus(to="~/Desktop")withopen(fp,encoding="utf-8")asf:corpus=[json.loads(l)forlinf]# Index and initiate the corpus as a concordancer objectC=Concordancer(corpus)C.set_cql_parameters(default_attr="word",max_quant=3)Interactive Search InterfaceYou can start an interactive server to query and read results through your browser:>>>server.run(C)Initializingserver...Startservingathttp://localhost:1420CQL Concordance searchAlternatively, you can work with theConcordancerobject, which allows you to send CQL queries to the corpus:cql='''verb:[pos="V.*"] noun:[pos="N[abch]"]'''concord_list=C.cql_search(cql,left=2,right=2)The results of a query is returned as a generator, which can be converted to a list of dictionaries (and then to JSON or other data structures for further uses):>>>concord_list=list(concord_list)>>>concord_list[:2][{'left':[{'word':'買','pos':'VC'},{'word':'了','pos':'Di'}],'keyword':[{'word':'覺得','pos':'VK'},{'word':'材質','pos':'Na'}],'right':[{'word':'很','pos':'Dfa'},{'word':'對','pos':'VH'}],'position':{'doc_idx':78,'sent_idx':13,'tk_idx':9},'captureGroups':{'verb':[{'word':'覺得','pos':'VK'}],'noun':[{'word':'材質','pos':'Na'}]}},{'left':[{'word':'“','pos':'PARENTHESISCATEGORY'},{'word':'不','pos':'D'}],'keyword':[{'word':'戴','pos':'VC'},{'word':'錶','pos':'Na'}],'right':[{'word':'世代','pos':'Na'},{'word':'”','pos':'VC'}],'position':{'doc_idx':52,'sent_idx':7,'tk_idx':36},'captureGroups':{'verb':[{'word':'戴','pos':'VC'}],'noun':[{'word':'錶','pos':'Na'}]}}]Keyword in ContextTo better read the concordance lines, passconcord_listintoconcordancer.kwic_print.KWIC()to print them as a keyword-in-context format in the console:>>>fromconcordancer.kwic_printimportKWIC>>>KWIC(concord_list[:5])leftkeywordrightLABEL:verbLABEL:noun-----------------------------------------------------------------------------------買/VC了/Di覺得/VK材質/Na很/Dfa對/VH覺得/VK材質/Na“/PARENTHESISCATEGORY不/D戴/VC錶/Na世代/Na”/VC戴/VC錶/Na聯名鞋/Na趁著/P過年/VA期間/Na穿出去/VB四處/D過年/VA期間/Na走/VA/WHITESPACE燒/VC錢/Na啊/T～/FW燒/VC錢/Na正/VH韓/Nc賣/VD家/Nc裡面/Ncd很/Dfa賣/VD家/NcSupported CQL featuresCQL search is supported throughcqls, which implements a (quite useful) subset of CQL:token:[],"我",[word="我"],[word!="我" & pos="N.*"]token-level quantifier:+,*,?,{n,m}grouping:("a" "b"? "c"){1,2}label:lab1:[word="我" & pos="N.*"] lab2:("a" "b")