Package: cordex
InstructionsNecessary files:corpus in conllu or TEI formatfile with structures definitionsPython 3.5+We suggest usage of pypy3 for faster processing.AboutThis library was developed to extract collocations from text in conllu or TEI format. Leveraging rules defined in the structure file (examples in tests/test_data/structures), it efficiently extracts and presents collocations.SetupYou can run the library using either python3 or pypy3. For a better experience, we recommend utilizing virtual environments.pipinstall-rrequirements.txtRunningExecution consists of three parts:setupexecutioncollecting resultsimportcordex# OPTIONAL: improve processing by downloading lookup lexicon (currently only available for Slovene)cordex.download('sl')# setupextractor=cordex.Pipeline("tests/test_data/structures/structures_UD.xml")# executionextraction=extractor("tests/test_data/input/ssj500k.small.conllu")# collecting resultsextraction.write("data/izhod.csv")SetupDuring this step you should provide processing settings. There is only one required parameter - path to structures file. Other parameters are optional.Exampleextractor=cordex.Pipeline("tests/test_data/structures/structures_UD.xml",statistics=False)ParametersstructuresRequired. Path to file with structures definitions. Examples of such file are available intests/test_data/structures.collocation_sentence_map_destDefault valueNone. Optional parameter, that should contain path to file or folder (it has to be of the same type aspathparameter inwrite()function). File or folder contain information about mappings between collocations and sentences. If this is set toNonemappings will not be stored. When thepathparameter inwrite()function leads to a file all mappings will also be stored in a file. Otherwise, program will create a directory at this destination and store mappings in multiple files inside this directory, one file per one syntactic structure. When you are usingget_listfunction you may set this toTrueand it will return mappings in a second element of a tuple. If the file or folder on a given location already exist, they will be deleted before processing.min_freqDefault value0. Number that indicate how many occurrences in corpus a collocation needs to be present in results.dbDefault valueNone. Path to interprocessing sqlite database file (if there is no file in that location it will create a new file). It enables us to process corpus in steps, and stores half processed data. This parameter is useful for processing bigger corpora, as a failsafe system. ValueNoneindicates that data will be stored only in memory.overwrite_dbDefault valueFalse. This parameter should be used together with parameterdb. WhenTrueit will overwrite old database file and start processing from the beginning.jos_msd_langDefault valueen. Set this toslwhen xpos tags are in Slovenian.ignore_punctuationsDefault valueFalse. When this isTrue, results containing punctuations will not be shown.fixed_restriction_orderDefault valueFalse. When this isTrue, results, where structure components are not in the same order as in structure definition files, will be ignored.lookup_lexiconDefault valueNone. Path to lookup lexicon. Lexicon is used to improve representations when JOS system is used. ValueNoneindicates that we are not using lookup lexicon.lookup_apiDefault valueFalse. When this isTrue, program will use api to improve representations when JOS system is used. Lookup lexicon will be ignored in this case.statisticsDefault valueTrue. Parameter that indicates whether we want statistics in output file or not.langDefault valuesl. Parameter that enables postprocessing for specific languages. Should contain lowercased 2-letter country abbreviation.jos_depparse_langDefault valuesl. When using JOS system, extraction will work with Slovenian (sl) or English (en) dependency parsing tags. This is not connected to UD dependency parsing in any way.ExecutionDuring this step extraction executes.Exampleextraction=extractor("tests/test_data/input/ssj500k.small.conllu")ParameterscorpusRequired. Path to corpus we want to be processed. Data may be in conllu or TEI format. This path should point to either concrete file containing corpus or directory containing multiple files in the same format. Examples of such files/folders are available intests/test_data/input.Collecting resultsWe support two ways of obtaining results. You may use functionwrite, to write results directly. You may also use functionget_list, which will return a list with results.ExamplesWrite example:extraction.write("data/izhod.csv")Get list example:results=extraction.get_list(sort_by=1,sort_reversed=True)ParameterspathRequired inwritefunction. Destination where results are going to be written. If this is a path to a folder, results are going to be stored in multiple files, otherwise in a single file.separatorDefault value\t. Optional parameter inwriteandget_listfunctions, that tells us what output files should be separated by. For.csvfiles shis should be,.decimal_separatorDefault value.. Optional parameter inwriteandget_listfunctions, that tells us what should be a separator for decimal numbers.sort_byDefault value-1. Parameter inwriteandget_listfunctions indicating by which column processing results will be sorted. Value-1indicates that results will be sorted by the order of structures given in structure definitions file.sort_reversedDefault valueFalse. This parameter is related tosort_by. When set toTrue, results will be sorted in reverse order of selected column.Optional: download lookup lexiconYou may download lookup lexicon, that enhances representations of collocations. By default, the lexicon will be downloaded to~/cordex_resourcesdirectory.
You may change this to other location by specifyingdirparameter in download function. Keep in mind that if custom path is selected, lexicon will only be used if an appropriatelookup_lexiconpath is given during Setup step.
If lexicon is on default path it will be used automatically. Lookup lexicon is currently only available for Slovenian language. You may download it using the following command. You may manually download lexicon fromclarin repository.cordex.download('sl')Instructions for running on big files (ie. Gigafida)You should run script using pypy3.Suggested running with saved mysql file in tmpfs. Instructions:sudomkdir/mnt/tmp
sudomount-ttmpfstmpfs/mnt/tmpIf running on big corpuses (ie. Gigafida have database in RAM):sudomkdir/mnt/tmp
sudomount-ttmpfstmpfs/mnt/tmp
sudomount-oremount,size=110G,noexec,nosuid,nodev,noatime/mnt/tmp