Package: pycldf
pycldfA python package to read and writeCLDFdatasets.InstallInstallpycldffromPyPI:pipinstallpycldfCommand line usageInstalling thepycldfpackage will also install a command line interfacecldf, which provides some sub-commands to manage CLDF datasets.Dataset discoverycldfsubcommands support dataset discovery as specified in thestandard.So a typical workflow involving a remote dataset could look as follows.Create a local directory to which to download the dataset (ideally including version info):$mkdirwacl-1.0.0Validating a dataset from Zenodo will implicitly download it, so running$cldfvalidatehttps://zenodo.org/record/7322688#rdf:ID=wacl--download-dirwacl-1.0.0/will download the dataset towacl-1.0.0.Subsequently we can access the data locally for better performance:$cldfstatswacl-1.0.0/#rdf:ID=wacl
<cldf:v1.0:StructureDatasetatwacl-1.0.0/cldf>value
--------------------------------------------------------------------------------------------
dc:bibliographicCitationHer,One-Soon,HaraldHammarströmandMarcAllassonnière-Tang.2022.
dc:conformsTohttp://cldf.clld.org/v1.0/terms.rdf#StructureDataset
dc:identifierhttps://wacl.clld.org
dc:licensehttps://creativecommons.org/licenses/by/4.0/
dc:sourcesources.bib
dc:titleWorldAtlasofClassifierLanguages
dcat:accessURLhttps://github.com/cldf-datasets/wacl
rdf:IDwacl
rdf:typehttp://www.w3.org/ns/dcat#DistributionTypeRows
----------------------------------
values.csvValueTable3338parameters.csvParameterTable1languages.csvLanguageTable3338codes.csvCodeTable2sources.bibSources2000(Note that locating datasets on Zenodo requires installation ofcldfzenodo.)Summary statistics$cldfstatsmydataset/Wordlist-metadata.json<cldf:v1.0:Wordlistatmydataset>

PathTypeRows
-------------------------------------
forms.csvFormTable1mydataset/sources.bibSources1ValidationArguably the most important functionality ofpycldfis validating CLDF datasets.By default, data files are read in strict-mode, i.e. invalid rows will result in an exception
being raised. To validate a data file, it can be read in validating-mode.For example the following output is generated$cldfvalidatemydataset/forms.csv
WARNINGforms.csv:duplicateprimarykey:(u'1',)WARNINGforms.csv:4:Sourcemissingsourcekey:Mei2005when reading the fileID,Language_ID,Parameter_ID,Value,Segments,Comment,Source
1,abcd1234,1277,word,,,Meier2005[3-7]
1,stan1295,1277,hand,,,Meier2005[3-7]
2,stan1295,1277,hand,,,Mei2005[3-7]Extracting human readable metadataThe information in a CLDF metadata file can be converted tomarkdown(a human readable markup language) runningcldfmarkdownPATH/TO/metadata.jsonA typical usage of this feature is to create aREADME.mdfor your dataset
(which, when uploaded to e.g. GitHub will be rendered nicely in the browser).Downloading media listed in a dataset's MediaTableTypically, CLDF datasets only reference media items. TheMediaTableprovides enough information, though,
to download and save an item's content. This can be done runningcldfdownloadmediaPATH/TO/metadata.jsonPATH/TO/DOWNLOAD/DIRTo minimize bandwidth usage, relevant items can be filtered by passing selection criteria in the formCOLUMN_NAME=SUBSTRINGas optional arguments. E.g. downloading could be limited to audio files passingMedia_Type=audio/(provided,Media_Typeis the name of the column withpropertyUrlhttp://cldf.clld.org/v1.0/terms.rdf#mediaType)Converting a CLDF dataset to an SQLite databaseA very useful feature of CSVW in general and CLDF in particular is that it
provides enough metadata for a set of CSV files to load them into a relational
database - including relations between tables. This can be done running thecldf createdbcommand:$cldfcreatedb-h
usage:cldfcreatedb[-h][--infer-primary-keys]DATASETSQLITE_DB_PATH

LoadaCLDFdatasetintoaSQLiteDB

positionalarguments:DATASETDatasetspecification(i.e.pathtoaCLDFmetadatafileortothedatafile)SQLITE_DB_PATHPathtotheSQLitedbfileFor a specification of the resulting database schema refer to the documentation insrc/pycldf/db.py.Python APIFor a detailed documentation of the Python API, refer to thedocs on ReadTheDocs.Reading CLDFAs an example, we'll read data fromWALS Online, v2020:>>>frompycldfimportDataset>>>wals2020=Dataset.from_metadata('https://raw.githubusercontent.com/cldf-datasets/wals/v2020/cldf/StructureDataset-metadata.json')For exploratory purposes, accessing a remote dataset over HTTP is fine. But for real analysis, you'd want to download
the datasets first and then access them locally, passing a local file path toDataset.from_metadata.Let's look at what we got:>>>print(wals2020)<cldf:v1.0:StructureDatasetathttps://raw.githubusercontent.com/cldf-datasets/wals/v2020/cldf/StructureDataset-metadata.json>>>>forcinwals2020.components:...print(c)...ValueTableParameterTableCodeTableLanguageTableExampleTableAs expected, we got aStructureDataset, and in
addition to the requiredValueTable, we also have a couple morecomponents.We can investigate the values usingpycldf's ORMfunctionality, i.e. mapping rows in the CLDF
data files to convenient python objects. (Take note of the limitations describe inorm.py, though.)>>>forvalueinwals2020.objects('ValueTable'):...break...>>>value<pycldf.orm.Valueid="81A-aab">>>>value.language<pycldf.orm.Languageid="aab">>>>value.language.cldfNamespace(glottocode=None,id='aab',iso639P3code=None,latitude=Decimal('-3.45'),longitude=Decimal('142.95'),macroarea=None,name='Arapesh (Abu)')>>>value.parameter<pycldf.orm.Parameterid="81A">>>>value.parameter.cldfNamespace(description=None,id='81A',name='Order of Subject, Object and Verb')>>>value.references(<ReferenceNekitel-1985[94]>,)>>>value.references[0]<ReferenceNekitel-1985[94]>>>>print(value.references[0].source.bibtex())@misc{Nekitel-1985,olac_field={syntax;general_linguistics;typology},school={AustralianNationalUniversity},title={SociolinguisticAspectsofAbu', a Papuan Language of the Sepik Area, Papua New Guinea},wals_code={aab},year={1985},author={Nekitel,OttoI.M.S.}}If performance is important, you can just read rows of data as pythondicts, in which case the references between
tables must be resolved "by hand":>>>params={r['id']:rforrinwals2020.iter_rows('ParameterTable','id','name')}>>>forvinwals2020.iter_rows('ValueTable','parameterReference'):...print(params[v['parameterReference']]['name'])...break...OrderofSubject,ObjectandVerbNote that we passed names of CLDF terms toDataset.iter_rows(e.g.id) specifying which columns we want to access
by CLDF term - rather than by the column names they are mapped to in the dataset.Writing CLDFWarning:Writing CLDF withpycldfdoes not automatically result in valid CLDF!
It does result in data that can be checked viacldf validate(seebelow),
though, so you should always validate after writing.frompycldfimportWordlist,Sourcedataset=Wordlist.in_dir('mydataset')dataset.add_sources(Source('book','Meier2005',author='Hans Meier',year='2005',title='The Book'))dataset.write(FormTable=[{'ID':'1','Form':'word','Language_ID':'abcd1234','Parameter_ID':'1277','Source':['Meier2005[3-7]'],}])results in$ ls -1 mydataset/
forms.csv
sources.bib
Wordlist-metadata.jsonmydataset/forms.csvID,Language_ID,Parameter_ID,Value,Segments,Comment,Source
1,abcd1234,1277,word,,,Meier2005[3-7]mydataset/sources.bib@book{Meier2005,author={Meier, Hans},year={2005},title={The Book}}mydataset/Wordlist-metadata.jsonAdvanced writingTo add predefined CLDF components to a dataset, use theadd_componentmethod:frompycldfimportStructureDataset,term_uridataset=StructureDataset.in_dir('mydataset')dataset.add_component('ParameterTable')dataset.write(ValueTable=[{'ID':'1','Language_ID':'abc','Parameter_ID':'1','Value':'x'}],ParameterTable=[{'ID':'1','Name':'Grammatical Feature'}])It is also possible to add generic tables:dataset.add_table('contributors.csv',term_uri('id'),term_uri('name'))which can also be linked to other tables:dataset.add_columns('ParameterTable','Contributor_ID')dataset.add_foreign_key('ParameterTable','Contributor_ID','contributors.csv','ID')Addressing tables and columnsTables in a dataset can be referenced using aDataset's__getitem__method,
passinga full CLDF Ontology URI for the corresponding component,the local name of the component in the CLDF Ontology,theurlof the table.Columns in a dataset can be referenced using aDataset's__getitem__method,
passing a tuple(<TABLE>, <COLUMN>)where<TABLE>specifies a table as explained
above and<COLUMN>isa full CLDF Ontolgy URI used aspropertyUrlof the column,thenameproperty of the column.See alsohttps://pycldf.readthedocs.io/en/latest/dataset.html#accessing-schema-objects-components-tables-columns-etcObject oriented access to CLDF dataThepycldf.ormmodule implements functionality
to access CLDF data via anORM.
Seehttps://pycldf.readthedocs.io/en/latest/orm.htmlfor
details.Accessing CLDF data via SQLThepycldf.dbmodule implements functionality
to load CLDF data into aSQLitedatabase. Seehttps://pycldf.readthedocs.io/en/latest/ext_sql.htmlfor details.See alsohttps://github.com/frictionlessdata/datapackage-py