Package: corpusinterface
Corpus InterfaceBasic functionality to maintain and load corpora.Installationpip install corpusinterfaceCorporaA number of are available in the two following config files:DCML_corpora.ini: contains corpora maintained by the DCML. Some of these are not publicly accessible so downloading them will fail. Please, contact us atdcml@epfl.chto request access.external_corpora.ini: contains external corpora that are not maintained by the DCML.Here are some references with more corpora (some of which are also in the config files from above):https://github.com/dharasim/MCR/wikihttps://github.com/ismir/mir-datasetsManaging CorporaAdding your own corpusTL;DRProvide a config fileyour-corpus.ini[Your Corpus]access:zipurl:http://your-website.com/your-corpus.zipand load it usinginit_config("your-corpus.ini").More detailsSay, you packaged a number of files into a corpusyour-corpus
  |- file_1.txt
  |- file_2.txt
  |- dir_1
    |- file_3.txt
    |- file_4.txtand let's assume you made it available as a zip archive athttp://your-website.com/your-corpus.zip(gitrepos andtar.gzfiles are also supported). Without a config file, this corpus can be added and accessed as follows:fromcorpusinterfaceimportconfig,load# add your corpusconfig.add_corpus("Your Corpus",access="zip",url="http://your-website.com/your-corpus.zip")# load the corpuscorpus=load("Your Corpus",download=True)# access the data (using a file_reader of your choice)forfileincorpus.data(file_reader=lambdafile,**kwargs:f"reading:{file}"):print(file)This will printreading: ~/corpora/Your Corpus/file_1.txt
reading: ~/corpora/Your Corpus/file_2.txt
reading: ~/corpora/Your Corpus/dir_1/file_3.txt
reading: ~/corpora/Your Corpus/dir_1/file_4.txtwith~being replaced with your home directory (paths might be displayed differently, depending on your operating system).Config files allow you to automate the procedure of adding a corpus and are convenient to provide more detailed information, in particular for other people who want to use your corpus.Config filesInstead of specifying the necessary information from within Python, you can also put it in a config file:[Your Corpus]access:zipurl:http://your-website.com/your-corpus.zipIf you put this file at the default location~/corpora/corpora.iniin your home directory or a filecorpora.iniin the current working directory, it is automatically loaded byinit_configon package import. Otherwise, you can load any config file by either callingreset_configconfig.reset_config("your-config-file.ini")which clears the config and reinitialises it, addingyour-config-file.ini(seeinit_configfor more fine-grained control) or by loading it separatelyconfig.load_config("your-config-file.ini")Default configA default config file is shipped with thecorpusinterfacepackage and automatically loaded byinit_config. It defines some useful defaults that are used for newly added corpora if no corpus-specific values are specified. You can see all the config information associated to your corpus by printing a summary:print(config.summary(corpus="Your Corpus"))[Your Corpus]access:zipurl:http://your-website.com/your-corpus.zipinfo:Noneroot:~/corporapath:~/corpora/Your Corpusparent:Noneloader:FileCorpusIn particular, the defaultrootdirectory~/corporawas added and the corpus is stored in apaththat is a subdirectory~/corpora/Your Corpusaccording to its name (more onrootandpathbelow). Moreover, by default we assume to have aFileCorpusconsisting of a simple collection of files.Special parametersThe parametersroot,path,parent,download,loader,access, andurlare special and their values are treated in a particular way.rootRoot directory to store the corpus in. This should be an absolute path,~is expanded to the user home. If a relative path is specified, a warning is issued and it is interpreted relative to the current working directory. Ifparentis non-empty, the value ofrootis ignored and instead the parent'spathis used. A call toconfig.get(Name, 'root')returns the effective value.pathDirectory to store the corpus in. This can bean absolute path (~is expanded to the user home), in which caserootis ignoreda relative path, in which case it is appended torootorbe empty, in which case the corpus[Name]is appended toroot.A call toconfig.get(Name, 'path')returns the effective value. Note that for sub-corpora (with non-emptyparent) the parent'spathis used instead ofroot.parentA parent corpus name or empty. If non-emtpy, the parent corpus should be defined separately and the value ofrootis ignored and replaced by the parent'spath.Initialisation (e.g. downloading fromurlwithaccessmethod) is delegated to the parent corpus when loading a sub-corpus.download,loader,access,urlSee the section on [loading a corpus](#Loading a corpus).Additional parametersYou can specify additional parameters that are handed over to the loader and (in case of theFileCorpusloader) further passed on the yourfile_readerfunction. For instance, you could specifyprefix:my prefixin the config file or equivalentlyconfig.add_corpus("Your Corpus",...,prefix="my prefix")from within Python. Your file reader can then make use of this parameter (provided as a keyword argument, so you have to refer to it by the correct name)file_reader=lambdafile,prefix,**kwargs:f"{prefix}:{file}"my prefix: ~/corpora/Your Corpus/file_1.txt
...This is also the reason why we always need**kwargsin a reader function to accept all keyword arguments that are provided, even if we decide to not use them.The config values can be dynamically overwritten in theloadfunctioncorpus=load("Your Corpus",...,prefix="other prefix")other prefix: ~/corpora/Your Corpus/file_1.txt
...or in thedatafunction:forfileincorpus.data(...,prefix="still different"):...still different: ~/corpora/Your Corpus/file_1.txt
...Controlling initialisationYou have full control over how the config is (re)initialised. A call toconfig.init_config()orconfig.reset_config()without any arguments will load the default config, look forcorpora.iniin~/corporaand the current working directory and load them, too, if present. This is equivalent to callingconfig.init_config(default=None,home=None,local=None)orconfig.reset_config(default=None,home=None,local=None)respectively. For each of these parameters you may alternatively specify a value ofTrue(meaning that youexpectthe respective config file to be loaded and otherwise an error is raised), orFalse(meaning that the respective config file isnotloaded, even if it exists). Additionally, you may specify one or more files that should additionally be loadedconfig.init_config("/path/to/file_1.ini","/path/to/file_2.ini",...)Loading a corpusCorpora are loaded with theloadfunctionfromcorpusinterfaceimportload# load the corpuscorpus=load("Your Corpus",download=True)Specifyingdownload=Trueindicates that the corpus should be downloaded if it cannot be found on disk. Theloadfunction looks up the given corpus in the config, retrieving any parameters (including default parameters) specified there. If you provide additional keyword arguments, these will overwrite parameters from the config with the same name. So you could, for instance, specify a different URL for downloadingcorpus=load("Your Corpus",url="some-other-url.com/corpus.zip"download=True)or a custom path for looking for the corpus on disk and/or downloading it tocorpus=load("Your Corpus",path="/my/custom/path/for/corpus/"download=True)Four parameters are processed by theloadfunction itself (download,access,url,loader).downloadandurlplay the obvious role described above.accessspecifies how the content should be accessed and together withurlis handled by thedownloadfunction (called byloadifdownload=Trueis specified).accesscan be a string ("git","zip", or"tar.gz") resulting in the corpus being downloaded and unpacked accordingly. It can also be a callable provided as a keyword argument toload. In that case the corpuspathis created on disk and the provided method is called with the corpus name and all keyword arguments, including any parameters specified in the config.Theloaderparameters is handled in a special way. If it is a callable, theloadfunction will ensure the corpus exists (potentially downloading it) and then call the specified method with all provided keyword arguments, including any parameters specified in the config. This means that you can simply specify any custom loader function you would like to usecorpus=load("Your Corpus",loader=my_special_loader_function)Ifloaderis a string,loadtries to look up the appropriate function in theloadersdictionary. So you can also add it there and request it by providing the corresponding string in theloadfunctionfromcorpusinterfaceimportload,loadersloaders["my custom loader"]=my_special_loader_functioncorpus=load("Your Corpus",loader="my custom loader")The advantage of this approach is that you can specify it in a config file so you don't need to pass it toloadeach timeloader:my custom loaderAdding the loader function can also be automised. For instance, if you have a special corpus type that you provide in a separate python module, you can simply add the loader function therefromcorpusinterfaceimportloadersclassMySpecialCorpus:...loaders["my custom loader"]=MySpecialCorpusGiven your custom config file, you corpus can then be loaded simply ascorpus=load("Your Corpus")without having to specify anything manually. Note that any loader function is provided with all keyword arguments, so it might be a good idea to use**kwargsto handle any unforeseen additional parameters, even if they are not used.FileCorpusThe default corpus type is defined by theFileCorpusclass. In a config file, it is specified byloader:FileCorpuswhich is the default if this parameter is not explicitly specified for a corpus. When callingload, they keyword argumentloader="FileCorpus"is looked up inloadersand the actualFileCorpusconstructor is called. In fact, the staticFileCorpus.initmethod is called to check for the mandatorypathargument and provide an interpretable error message if it is missing. TheFileCorpusclass assumes to find a collection of files atpathand makes them available via thefilesanddatamethod. Additionally, accepts four more parameters:file_regex: a regular expression for file names; if provided, files whose name doesnotmatch are ignoredpath_regex: a regular expression for paths; if provided, path (including the file name) that donotmatch are ignoredfile_exclude_regex: likefile_regexbutmatchesare ignorespath_exclude_regex: likepath_regexbutmatchesare ignoresAll additional keyword arguments are stored and passed on to calls ofdataandmetadata.filesThefilesfunction returns an iterator over files (after applying the*_regexexpressions, if provided). It returns their absolute paths.dataThedatafunction iterates overfilesand attempts to read them. If afile_readerfunction is provided as keyword argument upon initialisation or directly todata, it is called with the full path of the respective file as first argument and all keyword arguments. Otherwise (or iffile_reader=None)datareturns the absolute paths just likefiles.metadataThemetadatafunction looks for metadata in thepathlocation of the corpus. If ameta_readerfunction is provided as keyword argument upon initialisation or directly tometadata, it is called with the fullpathof the corpus as first argument and all keyword arguments. Otherwise (or ifmeta_reader=None)  the fullpathis returned.