Package: pyriksprot
Python package for reading and tagging Riksdagens ProtokollBatteries (tagger) not included.OverviewThis package is intended to cover the following use cases:Extract "text documents" from the Parla-CLARIN XML filesText can be extracted from the XML files at different granularity (paragraphs, utterance, speech, who, protocol). The text can be grouped (combined) into larger temporal blocks based on time (year, lustrum, decade or custom periods). Within each of these block the text in turn can be grouped by speaker attributes (who, party, gender).The text extraction can done using theriksprot2textutility, which is a CLI interface installed with the package, or in Python code using the API that this package exposes. The Python API exposed both streaming (SAX based) methods and a domain model API (i.e. Python classes representing protocols, speeches and utterances).Both the CLI and the API supports dehyphenation using method described inAnföranden: Annotated and Augmented Parliamentary Debates from Sweden, Stian Rødven Eide, 2020. The API also supports user defined text transformations.Extract PoS-tagged versions of the Parla-CLARIN XML filesPart-of-speech tagged versions of the protocols can be extracted with the same granularity and aggregation as described above for the raw text. The returned documents are tab-separated files with fields for text, baseform and pos-tag (UPOS, XPOS). Note that the actual part-of-speech tagging is done using tools found in thepyriksprot_taggingrepository (link).Currently there are no open-source tagged versions of the corpos avaliable. The tagging is done usingStanzawith Swedish language models produced and made publically avaliable by Språkbanken Text.Store extracted textThe extracted text can be stored as optionally compressed plain text files on disk, or in a ZIP-archive.Pre-requisitesPython >=3.11A folder containing the Riksdagen Protokoll (parliamentary protocols) Github repository.cdsome-folder\gitclone--branch"tag"tags/"tag"--depth1https://github.com/welfare-state-analytics/riksdagen-corpus.gitcdriksdagen-corpus
gitconfigcore.quotepathoffInstallation (Linux)Create an new isolated virtual environment for pyriksprot:mkdir/path/to/new/pyriksprot-foldercd/path/to/new/pyriksprot-folder
python-mvenv.venvActivate the environment:cd/path/to/new/pyriksprot-foldersource.venv/bin/activateInstallpyriksprotin activated virtual environment.pipinstallpyriksprotCLI riksprot2text:  Extract aggregated text corpus from Parla-CLARIN XML filesλriksprot2text--help

Usage:riksprot2text[OPTIONS]SOURCE_FOLDERTARGET

Options:-m,--mode[plain|zip|gzip|bz2|lzma]Targettype-t,--temporal-keyTEXTTemporalpartitionkey(s)-y,--yearsTEXTYearstoincludeinoutput-g,--group-keyTEXTPartitionkey(s)-p,--processesINTEGERRANGENumberofprocessestouse-l,--segment-level[protocol|speech|utterance|paragraph|who]Protocolextractsegmentlevel-e,--keep-orderKeepoutputinfilenameorder(slower,multiproc)-s,--skip-sizeINTEGERRANGESkipblocksofcharlengthlessthan-d,--dedentRemoveindentation-k,--dehyphenDehyphentext--helpShowthismessageandexit.λmetadata2db--help
Usage:metadata2db.py[OPTIONS]COMMAND[ARGS]...CLItooltomanageriksprotmetadata

Options:--helpShowthismessageandexit.

Commands:columnsdatabasedownloadfilenamesindexλmetadata2db.pydatabase--help
Usage:metadata2db.pydatabase[OPTIONS]TARGET

Options:--tagTEXTMetadataversion--source-folderTEXT--forceForceoverwrite--load-indexLoadutteranceindex--scripts-folderTEXTApplyscriptsinspecifiedfoldertoDB.IfnotspecifiedthescriptsareloadedfromSQL-module.--skip-scriptsSkiploadingSQLscripts--helpShowthismessageandexit.λmetadata2dbindex--help
Usage:metadata2db.pyindex[OPTIONS]CORPUS_FOLDERTARGET_FOLDER

Options:--helpShowthismessageandexit.Examples CLIAggregate text per year grouped by speaker. Store result in a single zip. Skip documents less than 50 characters.riksprot2text/path/to/corpusoutput.zip-mzip-tyear-lprotocol-gwho--skip-size50Aggregate text per decade grouped by speaker. Store result in a single zip. Remove indentations and hyphenations.riksprot2text/path/to/corpusoutput.zip-mzip-tdecade-lwho-gwho--dedent--dehyphenAggregate text using customized temporal periods and grouped by party.riksprot2text/path/to/corpusoutput.zip-mzip-t"1920-1938,1929-1945,1946-1989,1990-2020"-lwho-gpartyAggregate text per document and group by gender and party.riksprot2text/path/to/corpusoutput.zip-mzip-tprotocol-lwho-gparty-ggenderAggregate text per year grouped by gender and party and include only 1946-1989.riksprot2text/path/to/corpusoutput.zip-mzip-tyear-lwho-gparty-ggender-y1946-1989Python API - Iterate XML protocolsAggregate text per year grouped by speaker. Store result in a single zip. Skip documents less than 50 characters.importpyriksprottarget_filename:str=f'output.zip'opts={'source_folder':'/path/to/corpus','target':'outout.zip','target_type':'files-in-zip','segment_level':SegmentLevel.Who,'dedent':True,'dehyphen':False,'years':'1955-1965','temporal_key':TemporalKey.Protocol,'group_keys':(GroupingKey.Party,GroupingKey.Gender),}pyriksprot.extract_corpus_text(**opts)Iterate over protocol and speaker:frompyriksprotimportinterface,iterstorsitems:Iterable[interface.ProtocolSegment]=iterators.XmlProtocolTextIterator(filenames=filenames,segment_level=SegmentLevel.Who,segment_skip_size=0,processes=4)foriteminitems:print(item.who,len(item.text))Iterate over protocol and speech, skip empty:frompyriksprotimportinterface,iterstorsitems:Iterable[interface.ProtocolSegment]=iterators.XmlProtocolTextIterator(filenames=filenames,segment_level=SegmentLevel.Who,segment_skip_size=1,processes=4)foriteminitems:print(item.who,len(item.text))Iterate over protocol and speech, apply preprocess function(s):frompyriksprotimportinterface,iterstorsimportftfy# pip install ftfyimportunidecodefix_text:Callable[[str],str]=pyriksprot.compose([str.lower,pyriksprot.dedent,ftfy.fix_character_width,unidecode.unidecode])items:Iterable[interface.ProtocolSegment]=iterators.XmlProtocolTextIterator(filenames=filenames,segment_level=SegmentLevel.Speech,segment_skip_size=1,processes=4,preprocessor=fix_text,)foriteminitems:print(item.who,len(item.text))