Package: pylexibank
pylexibankpylexibankis a python package providing functionality to curate and aggregateLexibankdatasets.CompatibilityAt the core of the curation functionality provided bypylexibanklies integration
with the metadata catalogsGlottolog,ConcepticonandCLTS.
Not all releases of these catalogs are compatibly with all versions ofpylexibank.pylexibankGlottologConcepticonCLTS2.x>=4.x>=2.x1.x3.x>=4.x>=2.x>=2.xInstallSincepylexibankhas quite a few dependencies, installing it will result in installing
many other python packages along with it. To avoid any side effects for your default
python installation, we recommend installation in avirtual environment.Now you may installpylexibankvia pip or in development mode following the instructions
inCONTRIBUTING.md.Installingpylexibankwill also installcldfbench, which in turn installs a cli commandcldfbench. This command is used
to runpylexibankfunctionality from the command line as subcommands.cldfbenchis also used tomanage reference catalogs, in particular Glottolog,
Concepticon and CLTS. Thus, after installingpylexibankyou should runcldfbenchcatconfigto make sure the catalog data is locally available andpylexibankknows about it.Usagepylexibankcan be used in two ways:The command line interface provides mainly access to the functionality for thelexibankcuration workflow.Thepylexibankpackage can also be used like any other python package in your own
python code to access lexibank data in a programmatic (and consistent) way.Thecmd_makecldfmethodThe main goal ofpylexibankis creating high-quality CLDF Wordlists. This
happens in the customcmd_makecldfmethod of a Lexibank dataset. To make this task
easier,pylexibankprovidesaccess to Glottolog and Concepticon data:args.glottolog.apipoints to an instance ofCachingGlottologAPI(a subclass ofpyglottolog.Glottolog)args.concepticon.apipoints to an instance ofCachingConcepticonAPI(a subclass ofpyconcepticon.Concepticon)fine-grained control over form manipulationvia aDataset.form_spec, an instance
ofpylexibank.FormSpecwhich can be customized per
dataset.FormSpecis meant to capture the rules that have been used when compiling
the source data - for cases where the source data violates these rules, wholesale
replacement by listing a lexeme inetc/lexemes.csvis recommended.support for additional informationon lexemes, cognates, concepts and languages via
subclassing the defaults inpylexibank.modelseasy access to configuration datain a dataset'setc_dirsupport for segmentationusing thesegmentspackage with orthography profile(s):If an orthography profile is available asetc/orthography.tsv, asegments.Tokenizerinstance, initialized with this profile, will be available asDataset.tokenizerand automatically used byLexibankWriter.add_form.If a directoryetc/orthography/exists, all*.tsvfiles in it will be considered
orthography profiles, and adictmapping filename stem to tokenizer will be available. Tokenizer
selection can be controlled in two ways:Passing a keywordprofile=FILENAME_STEMinDataset.tokenizer()calls.Provide orthography profiles for each language and letDataset.tokenizerchose the tokenizer byitem['Language_ID'].Programmatic access to Lexibank datasetsWhile some level of support for reading and writing anyCLDFdataset is already provided by thepycldfpackage,pylexibank(building oncldfbench) adds another layer of abstraction which supportstreating Lexibank datasets as Python packages (and managing them viapip),a multi-step curation workflowaggregating collections of Lexibank datasets into a single SQLite database for efficient analysis.Installable andpylexibankenabled datasetsTurning a Lexibank dataset into a (pipinstallable) Python package is as simple as writing asetup scriptsetup.py.
But to make the dataset available for curation viapylexibank, the dataset must providea python modulecontaining a class derived frompylexibank.Dataset, which specifiesDataset.dir: A directory relative to which the thecuration directoriesare located.Dataset.id: An identifier of the dataset.which is advertised aslexibank.datasetentry pointinsetup.py. E.g.entry_points={'lexibank.dataset':['sohartmannchin=lexibank_sohartmannchin:Dataset',]},Turning datasets intopylexibankenabled python packages has multiple advantages:Datasets can be installed from various sources, e.g. GitHub repositories.Requirements files can be used "pin" particular versions of datasets for installation.Upon installation datasets can be discovered programmatically.Virtual environmentscan be used to manage projects which require different versions of the same dataset.ConventionsDataset identifier should be lowercase and either:the database name, if this name is established and well-known (e.g. "abvd", "asjp" etc),<author><languagegroup> (e.g. "grollemundbantu" etc)Datasets that require preprocessing with external programs (e.g. antiword, libreoffice) should store intermediate/artifacts in ./raw/ directory, and thecmd_installcode should install from that rather than requiring an external dependency.Declaring a dataset's dependence onpylexibank:specify minimum versions insetup.py, i.e. requirepylexibank>=1.x.yspecify exact versions in dataset'scldf-metadata.jsonusingprov:createdByproperty (pylexibankwill take care of this when the CLDF is created vialexibank makecldf).Datasets on GitHubGitHub provides a very good platform for collaborative curation of textual data
such as Lexibank datasets.Dataset curators are encouraged to make use of features in addition to the just version control, such asreleasesREADME.md, LICENSE, CONTRIBUTORS.mdNote that for datasets curated withpylexibank, summary statistics will be written toREADME.mdas part of themakecldfcommand.In addition to the support for collaboratively editing and versioning data, GitHub supports tying into additional services via webhooks. In particular, two of these services are relevant for Lexibank datasets:Continuous integration, e.g. via Travis-CIArchiving with Zenodo. Notes:When datasets are curated on GitHub and hooked up to ZENODO to trigger automatic deposits of releases, the release tagmuststart with a letter (otherwise the deposit will fail).Additional tags can be added to add context - e.g. when a release is triggered by a specific use case (for example the CLICS 2.0 release). This can be done usinggitas follows:gitcheckouttags/vX.Y.Z
gittag-a"clics-2.0"gitpushorigin--tagsAttributionThere are multiple levels of contributions to a Lexibank dataset:Typically, Lexibank datasets are derived from published data (be it supplemental material of a paper or public databases). Attribution to this source dataset is given by specifying its full citation in the dataset's metadata and by adding the source title to the release title of a lexibank dataset.Often the source dataset is also an aggregation of data from other sources. If possible, these sources (and the corresponding references) are kept in the Lexibank dataset's CLDF; otherwise we refer to the source dataset for a description of its sources.Deriving a Lexibank dataset from a source dataset using thepylexibankcuration workflow involves adding code, mapping to reference catalogs and to some extent also linguistic judgements. These contributions are listed in a dataset'sCONTRIBUTORS.mdand translate to the list of authors of released versions of the lexibank dataset.