Package: eis1600
EIS1600 ToolsFile PreparationProcessing WorkflowInstallationCommon Error MessagesSet UpWorking Directory StructureUsageconvert_mARkdown_to_EIS1600TMPids_insert_or_updatecheck_formattingreannotationq_tags_to_biomiu_random_revisionFile PreparationConvert from mARkdown to EIS1600TMP withconvert_mARkdown_to_EIS1600Check the.EIS1600TMPand correct tagged structureMark file as ready in the Google Spreadsheet (this includes the file into our processing pipeline)Optional: Runids_insert_or_updateon the checked.EIS1600TMP(or runincorporate_newly_prepared_files_in_corpuswhich will add IDs for all files listed as ready or double-checked).If you need to change the tagged structure in an.EIS1600file, you do those changes withSimple Markdown.
Runids_insert_or_updateto convert the changes inSimple MarkdowntoEIS1600 mARkdown.
Check the format of the EIS1600 file withcheck_formatting <path/to/file>.Processing WorkflowRunincorporate_newly_prepared_files_in_corpus. This script downloads the Google Sheet and processes all ready and double-checked files:Ready files are converted from EIS1600TMP to EIS1600 and IDs are added;Formatting of ready files (now EIS1600 files) and double-checked files is checked;IDs are updated if necessary.Files are now finalized and ready to be processed by the pipeline.Runanalyse_all_on_cluster. This script analysis all files prepared by the previous step:Each file is disassembled into MIUs;Analysis routine is run for each MIU;Results are returned as a JSON for each file that contains the annotated text, the populated yml and the analysis results (as df).The JSON files are ready to be imported into our database.InstallationYou can either do the complete local setup and have everything installed on your machine.
Alternatively, you can also use the docker image which can execute all the commands from the EIS1600-pkg.Docker InstallationInstall Docker Desktop:https://docs.docker.com/desktop/install/mac-install/It should install Docker Engine as well, which can be used through command line interface (CLI).To run a script from the EIS1600-pkg with docker, give the command to docker through CLI:$dockerrun<--gpusall>-it-v"</path/to/EIS1600>:/EIS1600"eis1600-pkg<EIS1600-pkg-commandanditsparams>Explanation:docker runstarts the image,-itpropagates CLI input to the image.--gpus all, optional to run docker with GPUs.-vwill virtualize a directory from your system in the docker image.-vvirtualized</path/to/EIS1600>from your system to/EIS1600in the docker image. You give the absolute path to ourEIS1600parent directory on your machine. Make sure to replace</path/to/EIS1600>with the correct path on your machine! This is the part in front of the colon, after the colon the destination inside the docker image is specified (this one is fixed).eis1600-pkgthe repository name on docker hub from where the image will be downloadedLast, the command from the package you want to execute including all parameters required by that command.E.G., to runq_tags_to_biofor toponym descriptions through docker:$dockerrun-it-v"</path/to/EIS1600>:/EIS1600"eis1600-pkgq_tags_to_bioTopo_Data/MIUs/TOPONYM_DESCRIPTION_DETECTION/toponym_description_training_dataTOPDTo run the annotation pipeline:$dockerrun--gpusall-it-v"</path/to/EIS1600>:/EIS1600"eis1600-pkganalyse_all_on_clusterMaybe add-Das parameter toanalyse_all_on_clusterbecause parallel processing does not work with GPU.Local SetupAfter creating and activating the eis16000_env (seeSet Up), use:$pipinstalleis1600In case you have an older version installed, use:$pipinstall--upgradeeis1600The package comes with different options, to install camel-tools use the following command.
Check also their installation instructions because atm they require additional packageshttps://camel-tools.readthedocs.io/en/latest/getting_started.html#installation$pipinstall'eis1600[NER]'If you want to run the annotation pipeline, you also need to download camel-tools data:$camel_data-idisambig-mle-calima-msa-r13To run the annotation pipeline with GPU, use this command:$pipinstall'eis1600[EIS]'Note. You can usepip freezeto check the versions of all installed packages, includingeis1600.Common Error MessagesYou need to download all the models ONE BY ONE from Google Drive.
Something breaks if you try to download the whole folder, and you get this error:OSError:Errornofilenamedpytorch_model.bin,tf_model.h5,model.ckpt.indexorflax_model.msgpackfoundindirectoryEIS1600_Pretrained_Models/camelbert-ca-finetunedBetter to syncEIS1600_Pretrained_Modelswith our nextcloud.If you want to installeis1600-pkgfrom source you have to add the data modules forgazetteersandhelpermanually.
You can find the modules in our nextcloud.Set Up Virtual Environment and Install the EIS1600 PKG thereTo not mess with other python installations, we recommend installing the package in a virual environment.
To create a new virtual environment with python, run:python3-mvenveis1600_envNB:while creating your new virtual environment, you must use Python 3.7 or 3.8, as these are version required by CAMeL-Tools.After creation of the environment it can be activated by:sourceeis1600_env/bin/activateThe environment is now activated and the eis1600 package can be installed into that environment with pip:$pipinstalleis1600This command installs all dependencies as well, so you should see lots of other libraries being installed. If you do not, you must have used a wrong version of Python while creating your virtual environment.You can now use the commands listed in this README.To use the environment, you have to activate it forevery session, by:sourceeis1600_env/bin/activateAfter successful activation, your user has the pre-text(eis1600_env).Probably, you want to create an alias for the source command in youraliasfile by adding the following line:aliaseis="source eis1600_env/bin/activate"Alias files:on Linux:.bash_aliasesOn Mac:.zshrcif you usezsh(default in the latest versions Mac OS);Structure of the working directoryThe working directory is always the mainEIS1600directory which is a parent to all the different repositories.
TheEIS1600directory has the following structure:|
|---| eis_env
|---| EIS1600_JSONs
|---| EIS1600_Pretrained_Models (for annotation, sync from Nextcloud)
|---| gazetteers
|---| Master_Chronicle
|---| OpenITI_EIS1600_Texts
|---| Training_DataPath variables are in the moduleeis1600/helper/repo.UsageAll commands must be run from theparent directoryEIS1600!See alsoProcessing Workflow.Use-Dflag to get detailed debug messages in the console.Annotation PipelineUse-Pflag to run annotation of MIUs in parallel, parallel processing will eat upALLresources!Use-Dflag to get detailed debug messages in the console.$analyse_all_on_clusterConvert mARkdown to EIS1600 filesConverts mARkdown file to EIS1600TMP (without inserting UIDs).
The .EIS1600TMP file will be created next to the .mARkdown file (you can input .inProcess or .completed files as well).
This command can be run from anywhere within the text repo - use auto complete (tab) to get the correct path to the file.
Alternative: open command line from the folder which contains the file which shall be converted.$convert_mARkdown_to_EIS1600TMP<uri>.mARkdownBatch processing of mARkdown filesRun from theparent directoryEIS1600.
Use the-eoption to convert all files from the EIS1600 repo.$convert_mARkdown_to_EIS1600-e<EIS1600_repo>EIS1600TMP to EIS1600EIS1600TMP files do not contain IDs yet, to insert IDs runids_insert_or_updateon the.EIS1600TMPfile.
Use auto complete (tab) to get the correct path to the file.$ids_insert_or_update<OpenITI_EIS1600_Text/data/path/to/file>.EIS1600TMPAdditionally, this routine updates IDs if you run it on a.EIS1600file.
Update IDs means inserting missing UIDs and updating SubIDs.$ids_insert_or_update<OpenITI_EIS1600_Text/data/path/to/file>.EIS1600Batch processingSee alsoProcessing Workflow.
Useincorporate_newly_prepared_files_in_corpusto add IDs to all ready files from the Google Sheet.$incorporate_newly_prepared_files_in_corpusCheck EIS1600 FormattingCheck if the formatting is correct (structural tagging)$check_formatting<OpenITI_EIS1600_Text/data/path/to/file>.EIS1600Batch processingCheck the formatting of all.EIS1600files:$check_formattingThis will create a log-file with all issues found. The log file is here:OpenITI_EIS1600_Texts/mal_formatted_texts.log.
It will print a list of files marked as 'ready' for which no.EIS1600TMPfile was found.
It will also print a list of files marked as 'double-cheked' for which no.EIS1600file was found.
Check if the author or book URI are still matching the folders and the file in theOpenITI_EIS1600_Textsdirectory.ReannotationThis script can be run on a folder containing files which were exported from the onine editor. Those files are MIUs but are missing directionality tags and paragraph tags (they use new lines to indicate paragraphs).
Use these flags to active the respective model for annotation:NERO [Onomastics]P [Persons and STFCOX]T [Toponyms and BDKOPRX]THIS WILL OVERWRITE THE ORIGINAL FILES IN THE FOLDER!$reannotation-NER-O-P-T<path/to/folder>Get training data from Q annotationsThis script can be used to transform Q-tags from EIS1600-mARkdown to BIO-labels.
The script will operate on a directory of MIUs and write a JSON file with annotated MIUs in BIO training format.
Parameters are:Path to directory containing annotated MIUs;Filename or path inside RESEARCH_DATA repo for JSON output fileBIO_main_class, optional, defaults to 'Q'. Try to use something more meaningful and distinguishable.$q_tags_to_bio<path/to/MIUs/><q_training_data><bio_main_class>For toponym definitions/descriptions:$q_tags_to_bioTopo_Data/MIUs/TOPONYM_DESCRIPTION_DETECTION/toponym_description_training_dataTOPDMIU revisionRun the following command from the root of the MIU repo to revise automated annotated files:$miu_random_revisionsWhen first run, the filefile_picker.ymlis added to the root of the MIU repository.
Make sure to specify your operating system and to set your initials and the path/command to/for Kate in this YAML file.system:...# options: mac, lin, win;reviewer:eis1600researcher# change this to your name;path_to_kate:kate# add absolute path to Kate on your machine; or a working alias (kate should already work)Optional, you can specify a path from where to open files - e.g. if you only want to open training-data, set:miu_main_path:./training_data/When revising files, remember to changereviewed:NOT REVIEWEDtoreviewed:REVIEWED