Package: rcsb.db
RCSB DBA collection of Python Database Utility ClassesIntroductionThis module contains a collection of utility classes for processing and loading PDB repository and
derived data content using relational and document database servers.  One target data store for
these tools is a document database used to exchange content within the RCSB PDB data pipeline.InstallationDownload the library source software from the project repository:gitclone--recurse-submoduleshttps://github.com/rcsb/py-rcsb_db.gitOptionally, run test suite (Python versions 2.7, 3.6, and 3.7) usingsetuptoolsortox:pythonsetup.pytestorsimplyrun

toxInstallation is via the programpip.  To run tests
from the source tree, the package must be installed in editable mode (i.e. -e):pipinstall-e.Installing in Ubuntu Linux (tested in 18.04)You will need a few packages, beforepip install .can work:sudoaptinstalldefault-libmysqlclient-devflexbisonInstalling on macOSTo use and develop this package on macOS requires a number of packages that are not
distributed as part of the base macOS operating system.
The following steps provide one approach to creating the development environment for this
package.  First, install the AppleXCodepackage and associate command-line tools.
This will provide essential compilers and supporting tools.  TheHomeBrewpackage
manager provides further access to a variety of common open source services and tools.
Follow the instructions provided by at theHomeBrewsite to
install this system.   Once HomeBrew is installed, you can further install theMariaDBandMongoDBpackages which
are required to support the ExDB  tools.  HomeBrew also provides a variety of options for
managing aPython virtual environments.Command Line InterfacesA convenience CLIschema_update_cliis provided for generating operational schema from
PDBx/mmCIF dictionary metadata.  Schema are encoded for the ExDB  API (rcsb), and
for the document schema encoded in JSON and BSON formats.  The latter schema can be used to
validate the loadable document objects produced for the collections served by MongoDB.=>schema_update_cli--help
usage:schema_update_cli[-h][--update_chem_comp_ref][--update_chem_comp_core_ref][--update_bird_chem_comp_ref][--update_bird_chem_comp_core_ref][--update_bird_ref][--update_bird_family_ref][--update_pdbx][--update_pdbx_core][--update_repository_holdings][--update_entity_sequence_clusters][--update_data_exchange][--update_ihm_dev][--update_drugbank_core][--update_config_all][--update_config_deployed][--update_config_test][--config_pathCONFIG_PATH][--config_nameCONFIG_NAME][--cache_pathSCHEMA_CACHE_PATH][--schema_typesSCHEMA_TYPES][--schema_levelsSCHEMA_LEVELS][--debug][--mock]optionalarguments:-h,--helpshowthishelpmessageandexit--update_chem_comp_refUpdateschemaforChemicalComponentreferencedefinitions--update_chem_comp_core_refUpdatecoreschemaforChemicalComponentreferencedefinitions--update_bird_chem_comp_refUpdateschemaforBirdChemicalComponentreferencedefinitions--update_bird_chem_comp_core_refUpdatecoreschemaforBirdChemicalComponentreferencedefinitions--update_bird_refUpdateschemaforBirdreferencedefinitions--update_bird_family_refUpdateschemaforBirdFamilyreferencedefinitions--update_pdbxUpdateschemaforPDBxentrydata--update_pdbx_coreUpdateschemaforPDBxcoreentry/entitydata--update_repository_holdingsUpdateschemaforrepositoryholdings--update_entity_sequence_clustersUpdateschemaforentitysequenceclusters--update_data_exchangeUpdateschemafordataexchangestatus--update_ihm_devUpdateschemaforI/HMdeventrydata--update_drugbank_coreUpdateDrugBankschema--update_config_allUpdateusingconfigurationsettings(e.g.DATABASE_NAMES_ALL)--update_config_deployedUpdateusingconfigurationsettings(e.g.DATABASE_NAMES_DEPLOYED)--update_config_testUpdateusingconfigurationsettings(e.g.DATABASE_NAMES_TEST)--config_pathCONFIG_PATHPathtoconfigurationoptionsfile--config_nameCONFIG_NAMEConfigurationsectionname--cache_pathCACHE_PATHSchemacachedirectorypath--schema_typesSCHEMA_TYPESSchemaencoding(rcsb|json|bson)(commaseparated)--schema_levelsSCHEMA_LEVELSSchemavalidationlevel(full|min)(commaseparated)--debugTurnonverboselogging--mockUseMOCKrepositoryconfigurationfordependenciesandtesting
________________________________________________________________________________For example, the following command will generate the JSON and BSON schema for the collections in the
pdbx_core schema.schema_update_cli--mock--schema_typesjson,bson\--schema_levelfull\--update_pdbx_core\--cache_path.\--config_path./rcsb/db/config/exdb-config-example.yml\--config_namesite_info_configurationA convenience CLIexdb_repo_load_cliis provided to support loading PDB repositories
containing entry and chemical reference data content types in the form of document collections
compatible with MongoDB.exdb_repo_load_cli--help

usage:exdb_repo_load_cli[-h][--full][--replace][--load_chem_comp_ref][--load_chem_comp_core_ref][--load_bird_chem_comp_ref][--load_bird_chem_comp_core_ref][--load_bird_ref][--load_bird_family_ref][--load_entry_data][--load_pdbx_core][--load_pdbx_core_merge][--load_pdbx_core_entry][--load_pdbx_core_entity][--load_pdbx_core_entity_monomer][--load_pdbx_core_assembly][--load_ihm_dev][--config_pathCONFIG_PATH][--config_nameCONFIG_NAME][--db_typeDB_TYPE][--document_styleDOCUMENT_STYLE][--read_back_check][--schema_levelSCHEMA_LEVEL][--load_file_list_pathLOAD_FILE_LIST_PATH][--fail_file_list_pathFAIL_FILE_LIST_PATH][--save_file_list_pathSAVE_FILE_LIST_PATH][--num_procNUM_PROC][--chunk_sizeCHUNK_SIZE][--file_limitFILE_LIMIT][--prune_document_sizePRUNE_DOCUMENT_SIZE][--debug][--mock][--cache_pathCACHE_PATH][--rebuild_cache][--rebuild_schema][--vrpt_repo_pathVRPT_REPO_PATH]optionalarguments:-h,--helpshowthishelpmessageandexit--fullFreshfullloadinanewtables/collections--replaceLoadwithreplacementinanexistingtable/collection(default)--load_chem_comp_refLoadChemicalComponentreferencedefinitions(publicsubset)--load_chem_comp_core_refLoadChemicalComponentCorereferencedefinitions(publicsubset)--load_bird_chem_comp_refLoadBirdChemicalComponentreferencedefinitions(publicsubset)--load_bird_chem_comp_core_refLoadBirdChemicalComponentCorereferencedefinitions(publicsubset)--load_bird_refLoadBirdreferencedefinitions(publicsubset)--load_bird_family_refLoadBirdFamilyreferencedefinitions(publicsubset)--load_entry_dataLoadPDBxentrydata(currentreleasedsubset)--load_pdbx_coreLoadallPDBxcorecollections(currentreleasedsubset)--load_pdbx_core_mergeLoadallPDBxcorecollectionswithmergedcontent(currentreleasedsubset)--load_pdbx_core_entryLoadPDBxcoreentry(currentreleasedsubset)--load_pdbx_core_entityLoadPDBxcoreentity(currentreleasedsubset)--load_pdbx_core_entity_monomerLoadPDBxcoreentitymonomer(currentreleasedsubset)--load_pdbx_core_assemblyLoadPDBxcoreassembly(currentreleasedsubset)--load_ihm_devLoadI/HMDEVmodeldata(currentreleasedsubset)--config_pathCONFIG_PATHPathtoconfigurationoptionsfile--config_nameCONFIG_NAMEConfigurationsectionname--db_typeDB_TYPEDatabaseservertype(default=mongo)--document_styleDOCUMENT_STYLEDocumentorganization(rowwise_by_name_with_cardinality|rowwise_by_name|columnwise_by_name|rowwise_by_id|rowwise_no_name--read_back_checkPerformreadbackcheckonalldocuments--schema_levelSCHEMA_LEVELSchemavalidationlevel(full|mindefault=None)--load_file_list_pathLOAD_FILE_LIST_PATHInputfilecontainingloadfilepathlist(overrideautomaticrepositoryscan)--fail_file_list_pathFAIL_FILE_LIST_PATHOutputfilecontainingfilepathsthatfailtoload--save_file_list_pathSAVE_FILE_LIST_PATHSaverepofilepathsfromautomaticfilesystemscaninthispath--num_procNUM_PROCNumberofprocessestoexecute(default=2)--chunk_sizeCHUNK_SIZENumberoffilesloadedperprocess--file_limitFILE_LIMITLoadfilelimitfortesting--prune_document_sizePRUNE_DOCUMENT_SIZEPrunelargedocumentstothissizelimit(MB)--debugTurnonverboselogging--mockUseMOCKrepositoryconfigurationfortesting--cache_pathCACHE_PATHCachepathforresourcefiles--rebuild_cacheRebuildcachedresourcefiles--rebuild_schemaRebuildschemaon-the-flyifnotcached--vrpt_repo_pathVRPT_REPO_PATHPathtovalidationreportrepository
________________________________________________________________________________Part of the schema definition process supported by this module involves refining
the dictionary metadata with more specific data typing and coverage details.
A scanning tools is provided to collect and organize these details for the
other ETL tools in this package.  The following convenience CLI,repo_scan_cli,
is provided to scan supported PDB repository content and update data type and coverage details.repo_scan_cli--help

usage:repo_scan_cli[-h][--scanTypeSCANTYPE][--scan_chem_comp_ref|--scan_chem_comp_core_ref|--scan_bird_chem_comp_ref|--scan_bird_chem_comp_core_ref|--scan_bird_ref|--scan_bird_family_ref|--scan_entry_data|--scan_ihm_dev][--config_pathCONFIG_PATH][--config_nameCONFIG_NAME][--input_file_list_pathINPUT_FILE_LIST_PATH][--output_file_list_pathOUTPUT_FILE_LIST_PATH][--fail_file_list_pathFAIL_FILE_LIST_PATH][--scan_data_file_pathSCAN_DATA_FILE_PATH][--coverage_file_pathCOVERAGE_FILE_PATH][--type_map_file_pathTYPE_MAP_FILE_PATH][--num_procNUM_PROC][--chunk_sizeCHUNK_SIZE][--file_limitFILE_LIMIT][--debug][--mock][--working_pathWORKING_PATH]optionalarguments:-h,--helpshowthishelpmessageandexit--scanTypeSCANTYPERepositoryscantype(full|incr)--scan_chem_comp_refScanChemicalComponentreferencedefinitions(publicsubset)--scan_chem_comp_core_refScanChemicalComponentCorereferencedefinitions(publicsubset)--scan_bird_chem_comp_refScanBirdChemicalComponentreferencedefinitions(publicsubset)--scan_bird_chem_comp_core_refScanBirdChemicalComponentCorereferencedefinitions(publicsubset)--scan_bird_refScanBirdreferencedefinitions(publicsubset)--scan_bird_family_refScanBirdFamilyreferencedefinitions(publicsubset)--scan_entry_dataScanPDBentrydata(currentreleasedsubset)--scan_ihm_devScanPDBDEVI/HMentrydata(currentreleasedsubset)--config_pathCONFIG_PATHPathtoconfigurationoptionsfile--config_nameCONFIG_NAMEConfigurationsectionname--input_file_list_pathINPUT_FILE_LIST_PATHInputfilecontainingfilepathstoscan--output_file_list_pathOUTPUT_FILE_LIST_PATHOutputfilecontainingfilepathsscanned--fail_file_list_pathFAIL_FILE_LIST_PATHOutputfilecontainingfilepathsthatfailscan--scan_data_file_pathSCAN_DATA_FILE_PATHOutputworkingfilestoringscandata(Pickle)--coverage_file_pathCOVERAGE_FILE_PATHCoveragemap(JSON)outputpath--type_map_file_pathTYPE_MAP_FILE_PATHTypemap(JSON)outputpath--num_procNUM_PROCNumberofprocessestoexecute(default=2)--chunk_sizeCHUNK_SIZENumberoffilesloadedperprocess--file_limitFILE_LIMITLoadfilelimitfortesting--debugTurnonverboselogging--mockUseMOCKrepositoryconfigurationfortesting--working_pathWORKING_PATHWorkingpathfortemporaryfiles
________________________________________________________________________________The following CLI provides a preliminary access to ETL functions for processing
derived content types such as sequence comparative data.etl_exec_cli--help
usage:etl_exec_cli[-h][--full][--etl_entity_sequence_clusters][--etl_repository_holdings][--data_set_idDATA_SET_ID][--sequence_cluster_data_pathSEQUENCE_CLUSTER_DATA_PATH][--sandbox_data_pathSANDBOX_DATA_PATH][--config_pathCONFIG_PATH][--config_nameCONFIG_NAME][--db_typeDB_TYPE][--read_back_check][--num_procNUM_PROC][--chunk_sizeCHUNK_SIZE][--document_limitDOCUMENT_LIMIT][--prune_document_sizePRUNE_DOCUMENT_SIZE][--debug][--mock][--cache_pathCACHE_PATH][--rebuild_cache]optionalarguments:-h,--helpshowthishelpmessageandexit--fullFreshfullloadinanewtables/collections(Default)--etl_entity_sequence_clustersETLentitysequenceclusters--etl_repository_holdingsETLrepositoryholdings--data_set_idDATA_SET_IDDatasetidentifier(default=2018_14forcurrentweek)--sequence_cluster_data_pathSEQUENCE_CLUSTER_DATA_PATHSequenceclusterdatapath(defaultsetbyconfiguration--sandbox_data_pathSANDBOX_DATA_PATHDateexchangesandboxPathdatapath(defaultsetbyconfiguration--config_pathCONFIG_PATHPathtoconfigurationoptionsfile--config_nameCONFIG_NAMEConfigurationsectionname--db_typeDB_TYPEDatabaseservertype(default=mongo)--read_back_checkPerformreadbackcheckonalldocuments--num_procNUM_PROCNumberofprocessestoexecute(default=2)--chunk_sizeCHUNK_SIZENumberoffilesloadedperprocess--document_limitDOCUMENT_LIMITLoaddocumentlimitfortesting--prune_document_sizePRUNE_DOCUMENT_SIZEPrunelargedocumentstothissizelimit(MB)--debugTurnonverboselogging--mockUseMOCKrepositoryconfigurationfortesting--cache_pathCACHE_PATHPathcontainingcachedirectories--rebuild_cacheRebuildcachedresourcefiles
________________________________________________________________________________ExamplesIf you are working in the source repository, then you can run the CLI commands in the following manner.
The following examples load data in the mock repositories in source distribution assuming you have a local
default installation of MongoDb (no user/pw assigned).To run the command-line interfaceexdb_repo_load_clioutside of the source distribution, you will need to
create a configuration file with the appropriate path details and authentication credentials.For instance, to perform a fresh/full load of all of the chemical component definitions in the mock repository:cdrcsb/db/scripts
pythonRepoLoadExec.py--full--load_chem_comp_ref\--config_path../config/exdb-config-example.yml\--config_namesite_info_configuration\--fail_file_list_pathfailed-cc-path-list.txt\--read_back_checkThe following illustrates, a full load of the mock structure data repository followed by a reload with replacement of
this same data.cdrcsb/db/scripts
pythonRepoLoadExec.py--mock--full--load_entry_data\--config_path../config/exdb-config-example.yml\--config_namesite_info_configuration\--save_file_list_pathLATEST_PDBX_LOAD_LIST.txt\--fail_file_list_pathfailed-entry-path-list.txt

pythonRepoLoadExec.py--mock--replace--load_entry_data\--config_path../config/exdb-config-example.yml\--config_namesite_info_configuration\--load_file_list_pathLATEST_PDBX_LOAD_LIST.txt\--fail_file_list_pathfailed-entry-path-list.txtConfiguration ExampleRCSB/PDB repository path details are stored as configuration options.
An example configuration file included in this package is shown below.
This example is references dictionary resources and mock repository data
provided in the package inrcsb/mock-data/*. Example configuration details are
stored in rcsb/db/config/exdb-config-example.yml.Thesite_info_configurationsection
in this file provides database server connection details and common path details.
This is followed by sections specifying the dictionaries, helper functions, and
configuration used to define the schema for the each supported content type
(e.g., pdbx_core, chem_comp_core, bird_chem_comp_core,.. ).site_info_configuration:# Site specific path and server configuration options - (REFERENCING DEVELOPMENT RESOURCES)#CONFIG_SUPPORT_TOKEN:CONFIG_SUPPORT_TOKEN_ENV## Database server connection details#MONGO_DB_HOST:localhostMONGO_DB_PORT:"27017"_MONGO_DB_USER_NAME:""_MONGO_DB_PASSWORD:""MYSQL_DB_HOST_NAME:localhostMYSQL_DB_PORT_NUMBER:"3306"_MYSQL_DB_USER_NAME:wrIzBGtCsQmkjc7tbEPQ3oEaOnpvivXaKcQsvXD6kn4KHMvA7LCL4O9GlAI=_MYSQL_DB_PASSWORD:qXPp32Z6DhNVMwo9fQIK5+KB13c1Jd43E3Bn6LmJcSyXc0NAt4H/hwo/xglYpmELV5Vqaw==_MYSQL_DB_PASSWORD_ALT:s6mNxq3FIwZLrLiIeHpDZQcuVxfQqrR3gA+dEMOGgHwsjrJV5da08H74RmnNRus74Q==MYSQL_DB_DATABASE_NAME:mysqlCRATE_DB_HOST:localhostCRATE_DB_PORT:"4200"COCKROACH_DB_HOST:localhostCOCKROACH_DB_PORT:"26257"COCKROACH_DB_NAME:system_COCKROACH_DB_USER_NAME:HR2ez8iLbEpvN+hXKIQS3qa6/QpiFRpf/WvrfHiwfjcL09E+iWTQJhsxTsw=## Primary repository data and related computed repository data paths#BIRD_REPO_PATH:MOCK_BIRD_REPOBIRD_FAMILY_REPO_PATH:MOCK_BIRD_FAMILY_REPOBIRD_CHEM_COMP_REPO_PATH:MOCK_BIRD_CC_REPOCHEM_COMP_REPO_PATH:MOCK_CHEM_COMP_REPOPDBX_REPO_PATH:MOCK_PDBX_SANDBOXRCSB_EXCHANGE_SANDBOX_PATH:MOCK_EXCHANGE_SANDBOXIHM_DEV_REPO_PATH:MOCK_IHM_REPOVRPT_REPO_PATH:MOCK_VALIDATION_REPORTSVRPT_REPO_PATH_ENV:VRPT_REPO_PATH_ALT#RCSB_EDMAP_LIST_PATH:MOCK_EXCHANGE_SANDBOX/status/edmaps.json#RCSB_SEQUENCE_CLUSTER_DATA_PATH:cluster_data/mmseqs_clusters_currentSIFTS_SUMMARY_DATA_PATH:sifts-summary# -------------------------------------------------------------------------------------------#   -- Below are common across current deployments -## Supporting and integrated resource data cache directory names##DRUGBANK_CACHE_DIR: DrugBank_DRUGBANK_AUTH_USERNAME:0qrpNd4OhGuVsJqEpcsAVEovZ0hl6QkgxmbTy3bPssd06Z9tuM6bJqgsWwmFCd0JnjIMIEWyKPMmF1pI5g==_DRUGBANK_AUTH_PASSWORD:lA/K132i8DOtLdMHfm3gpNqprZ6ABKjsCRxfcXIMnxpKQzBv/B6dmC7x1vRO86JhqdT0b84=DRUGBANK_MOCK_URL_TARGET:DrugBank/full_database.zip##ATC_CACHE_DIR: atc#CHEM_COMP_CACHE_DIR: chem_compNCBI_TAXONOMY_CACHE_DIR:NCBIENZYME_CLASSIFICATION_CACHE_DIR:ecSTRUCT_DOMAIN_CLASSIFICATION_CACHE_DIR:domains_structSIFTS_SUMMARY_CACHE_DIR:sifts_summaryDICTIONARY_CACHE_DIR:dictionariesDATA_TYPE_INFO_CACHE_DIR:data_type_and_coverageREPO_UTIL_CACHE_DIR:repo_utilEXDB_CACHE_DIR:exdbCITATION_REFERENCE_CACHE_DIR:cit_ref##PROVENANCE_INFO_LOCATOR:https://raw.githubusercontent.com/rcsb/py-rcsb_exdb_assets/master/provenance/rcsb_extend_provenance_info.jsonPROVENANCE_INFO_CACHE_DIR:provenance#SCHEMA_DEFINITION_LOCATOR_PATH:https://raw.githubusercontent.com/rcsb/py-rcsb_exdb_assets/master/schema_definitionsSCHEMA_DEFINITION_CACHE_DIR:schema_definitionsJSON_SCHEMA_DEFINITION_LOCATOR_PATH:https://raw.githubusercontent.com/rcsb/py-rcsb_exdb_assets/master/json_schema_definitionsJSON_SCHEMA_DEFINITION_CACHE_DIR:json_schema_definitions## Helper class binding and mappings#DICT_METHOD_HELPER_MODULE_PATH_MAP:rcsb.utils.dictionary.DictMethodEntryHelper:rcsb.utils.dictionary.DictMethodEntryHelperrcsb.utils.dictionary.DictMethodChemRefHelper:rcsb.utils.dictionary.DictMethodChemRefHelperrcsb.utils.dictionary.DictMethodEntityHelper:rcsb.utils.dictionary.DictMethodEntityHelperrcsb.utils.dictionary.DictMethodAssemblyHelper:rcsb.utils.dictionary.DictMethodAssemblyHelperrcsb.utils.dictionary.DictMethodEntityInstanceHelper:rcsb.utils.dictionary.DictMethodEntityInstanceHelper# ------ ------ ------ ------ ------ ------ ------ -------# ADDED rcsb.db V0.966 Source dictionary locators -#PDBX_DICT_LOCATOR:https://raw.githubusercontent.com/rcsb/py-rcsb_exdb_assets/master/dictionary_files/reference/mmcif_pdbx_v5_next.dicRCSB_DICT_LOCATOR:https://raw.githubusercontent.com/rcsb/py-rcsb_exdb_assets/master/dictionary_files/dist/rcsb_mmcif_ext.dicIHMDEV_DICT_LOCATOR:https://raw.githubusercontent.com/rcsb/py-rcsb_exdb_assets/master/dictionary_files/reference/ihm-extension.dicFLR_DICT_LOCATOR:https://raw.githubusercontent.com/rcsb/py-rcsb_exdb_assets/master/dictionary_files/reference/flr-extension.dicVRPT_DICT_LOCATOR:https://raw.githubusercontent.com/rcsb/py-rcsb_exdb_assets/master/dictionary_files/reference/vrpt_mmcif_ext.dicVRPT_DICT_MAPPING_LOCATOR:https://raw.githubusercontent.com/rcsb/py-rcsb_exdb_assets/master/dictionary_files/reference/vrpt_dictmap.json# ------ ------ ------ ------ ------ ------ ------ ------ ------# Added in rcsb.db V0.966 - Data type details and type mappingAPP_DATA_TYPE_INFO_LOCATOR:https://raw.githubusercontent.com/rcsb/py-rcsb_exdb_assets/master/data_type_and_coverage/app_data_type_mapping.cifINSTANCE_DATA_TYPE_INFO_LOCATOR_PATH:https://raw.githubusercontent.com/rcsb/py-rcsb_exdb_assets/master/data_type_and_coverage#CONTENT_DEF_HELPER_MODULE:rcsb.db.helpers.ContentDefinitionHelperDOCUMENT_DEF_HELPER_MODULE:rcsb.db.helpers.DocumentDefinitionHelperCONFIG_APPEND_LOCATOR_PATHS:-https://raw.githubusercontent.com/rcsb/py-rcsb_exdb_assets/master/config/exdb-config-schema.yml# ------ ------ ------ ------ ------ ------ ------ ------ ------#  Added V1.001 for stash storage server#  -- This is a placeholder configuration to support remote testing --#  local|serverSTASH_MODE:localSTASH_LOCAL_BASE_PATH:stash-storage#STASH_SERVER_URL:https://raw.githubusercontent.comSTASH_SERVER_FALLBACK_URL:https://raw.githubusercontent.com_STASH_SERVER_BASE_PATH:bIo7kGc2w6Oel0QNr7Pc/4bFfDQayhPxddnGHynP6yudxc44QYuAFoTFdOqY2ZzsM2DEk56r26MfG66bEQ42lp38guy837xwzN1Vgu+r9zvAm11HXEA=REFERENCE_SEQUENCE_ALIGNMETS:PDB##