Package: pyriksprot-tagger
Riksdagens Protokoll Part-Of-Speech TaggingThis package implements part-of-speech tagging ofRiksdagens Protokoll(Parla-CLARIN)[https://clarin-eric.github.io/parla-clarin/] files.PrerequisitesThe workflow makes use ofGnu make,git,pyenvandpoetry.Latest version ofwelfare-state-analytics/pyriksprot.Latest version ofwelfare-state-analytics/pyriksprot_tagger.NLP models forSparv and Stanzainstalled.A local copy ofriksdagen-corpusGithub repository.Tagging WorkflowUpdatepyriksprot-taggerconfigurationfile (.env).Updateriksdagen-corpusrepository.Run thetag.shscript:PYTHONPATH=.nohup./tag.sh--target-folder/path/to/output/data>tag-it.version.log&You can also execute a predefined make recepi:maketag-itIf you runtag.shwithout parameters then the values found in.envwill be used. You can also specify
parameters as command line options:usage: ./tag.sh [--data-folder folder] [--source-pattern pattern] --target-folder folder --tag tag [--force] [--update] [--max-procs n]]
Creates new database using source as template. Source defaults to production.

   --data-folder             source root folder
   --source-pattern          source folder pattern
   --target-folder           target folder
   --tag                     source corpus tag
   --force                   drop target if exists
   --update                  update target if exists
   --max-procs               max number of parallel jobsNote thattag.shwill raise an error if the checkout tag in the Git repository and tag specified in .env (or as a parameter) mismatch.Metadata WorkflowThis workflow processes the corpus metadata and generates an Sqlite relational database. This database is used by the Westac Notebooks when filtering and pivoting data based on speaker, party etc. Usewelfare-state-analytics/pyriksprotto create or update the metadata:Updatepyriksprot/.envand set current tag.Run themake metadatato create a metadata database for current tag:Detailed workflowDue to potentiallyy breaking changes in the metadata we need to find differences between the new and old version of the metadata. If new fields or coded values have been added or change, or any other breaking change has been made then most likely the scripts that processes the metadata needs to be updated. Data updates are made both using SQL scripts and Python scripts.Identify breaking changes.Download previous and current metadata in two seperate folders:metadata2db download v0.9.0 ./tmp/metadata/v0.9.0
metadata2db download v0.10.0 ./tmp/metadata/v0.10.0💡 Alt:python pyriksprot/scripts/metadata2db.py download v0.10.0 ./tmp/metadata/v0.10.0💡 Usemoshfeu.compare-foldersto compare folders in vscode.If you find structural differences than you need to file an issue and request the system to be updated to deal with the changes. Modulepyriksprot.sqlcontains SQL scripts for metadata schema and (some) updates. Furthermore, some schema changes need to be handled in thepyriksprot.modulemodule (e.g.pyriksprot.module.config). Changes may of course also affect thepenelopecorpus pipeline.Create a metadata database usingwelfare-state-analytics/pyriksprotfor given tag:Updatepyriksprot/.env(e.g. tag)Run themetadatarecipe:makemetadataSpeech Corpus WorkflowCreate a default speech corpus usingwelfare-state-analytics/pyriksprot_taggerfor given tag:Run te recipiextract-speeches-to-feather:makeextract-speeches-to-featherSee appendix below if you instead want to usesnakemakefor updating repository and tagging,InstallpyriksprottaggerEasiest way is to clone the GitHub repository:cd/path/to/any/folder
gitclonegit@github.com:welfare-state-analytics/pyriksprot_taggercdpyriksprot_tagger
pyenvlocal3.11.3
poetryshell
pipinstalltorch
poetryinstallYou can alsoinstall the taggerin an isolated Python virtual environment. This method requires you to manually download certain scripts depending on your specific workflow.Install Sparv and Stanza modelsUsestanza-models.shscript to download Stanza files. Note that the target folder specified in the script must be the same as the folder specified by the STANZA_DATADIR environment variable (in .env).Optional: Usepenelope/scripts/install-spacy-models.shto install relevant SpaCy models.Update configurationUpdate or create dotenv (.env) in thepyriksprot_taggerfolder with the following variable definitions:Environment variableDescriptionRIKSPROT_DATA_FOLDERParent folder (location) of Riksdagens corpus data folderRIKSPROT_REPOSITORY_URLhttps://github.com/welfare-state-analytics/riksdagen-corpus.gitRIKSPROT_REPOSITORY_TAGTarget corpus version. Must be a valid Github tagSPARV_DATADIRSparv data folderSTANZA_DATADIRStanza data folderRIKSPROT_DATA_FOLDER="/path/to/data/folder"RIKSPROT_REPOSITORY_URL="https://github.com/welfare-state-analytics/riksdagen-corpus.git"RIKSPROT_REPOSITORY_TAG="vx.y.z"SPARV_DATADIR="/path/to/sparv_datadir"STANZA_DATADIR="/path/to/stanza_datadir"AppendixSetup a local copy of riksdagen-corpus Github repositoryIfriksdagen-corpusrepository folder already exists, then do an update:cd/path/to/git/repository
gitpullIf repository folder doesn't exist:cd/path/to/parent-folder
gitclonegit@github.com:welfare-state-analytics/pyriksprot_tagger.gitYou need to checkout the specific tag that you want to process:cd/path/to/git/repository
gitcheckoutvx.y.zMake sure to update file timestamps to latest commit timestamp!cd/path/to/pyriksprot-tagger
./pyriksprot_tagger/scripts/update-timestampsInstall pyriksprot-tagger from PyPIVerify current Python version (pyenvis recommended for easy switch between versions).Create a new Python virtual environment (sandbox):cd/some/folder
mkdirriksprot_taggingcdriksprot_tagging
python-mvenv.venvsource.venv/bin/activateInstall the pipeline and run setup script.pipinstallpyriksprot_tagger
setup-pipelineTo tag protocols you first need to activate the installed environment, and then follow steps above on how to tag protocols using snakemake.cd/some/folder/pyriksprotsource.venv/bin/activateCreate or update the repository using snakemake (not recommended)This is an alternative way of updating the corpus repository.%cd/path/to/pyriksprot-tagger/folderIf you want to create a new clone of the repository:%makefull-clone-repositoryIf you want to update an existing repository:%makefull-pull-repositoryIf you want to save space and do a shallow clone%makeshallow-update-repositoryUpdate timestamp of repository work folder files to match last commit timestamp. Important! This isrequiredif you use Snakemake when tagging:%makeupdate-repository-timestampsHow to annotate protocols using snakemake (not recommended)Annotate using default settings:makeannotateAnnotate a single year (and set cpu count).makeannotateYEAR=1960CPU_COUNT=1Call snakemake directly:nohupmakeannotatePROCESSES_COUNT=4>&run.log&ornohuppoetryrunsnakemake--config-j4--keep-going--keep-target-files&