Package: data-dictionary-cui-mapping
data-dictionary-cui-mappingThis package assists with mapping a user's data dictionary fields toUMLSconcepts. It is designed to be modular and flexible to allow for different configurations and use cases.Roughly, the high-level steps are as follows:Configure yaml filesLoad in data dictionaryPreprocess desired columnsQuery for UMLS concepts using any or all of the following pipeline modules:umls(UMLS API)metamap(MetaMap API)semantic_search(relies on access to a custom Pinecone vector database)hydra_search(combines any combination of the above three modules)Manually curate/select concepts in excelCreate data dictionary file with new UMLS concept fieldsPrerequisitesFor UMLS API and MetaMap API, you will need to have an account with the UMLS API and/or MetaMap API. You can sign up for an account here:https://www.nlm.nih.gov/research/umls/index.htmlFor Semantic Search with Pinecone, you will need to have an account with Pinecone. You can sign up for an account here:https://www.pinecone.io/. Please reach out to me if you would like temporary access to my Pinecone index to explore these embeddings.InstallationUse the package managerpipto installdata-dictionary-cui-mappingfrom PyPI or pip install from theGitHub repo. The project usespoetryfor packaging and dependency management.pipinstalldata-dictionary-cui-mapping#pip install git+https://github.com/kevon217/data-dictionary-cui-mapping.gitInput: Data DictionaryBelow is a sample data dictionary format (.csv) that can be used as input for this package:variable nametitlepermissible value descriptionsAgeYrsAge in yearsCaseContrlIndCase control indicatorCase;Control;UnknownConfiguration FilesIn order to run and customize these pipelines, you will need to create/edit yaml configuration files located in configs. Run configurations are saved and can be reloaded.├───ddcuimap
│├───configs
│││config.yaml
│││__init__.py
│││
││├───apis
│││__init__.py
│││config_metamap_api.yaml
│││config_pinecone_api.yaml
│││config_umls_api.yaml
│││
││├───custom
│││de.yaml
│││hydra_base.yaml
│││pvd.yaml
│││title_def.yaml
│││
││├───semantic_search
│││embeddings.yamlCUI Batch Query PipelinesSTEP-1A: RUN BATCH QUERY PIPELINEIMPORT PACKAGES# from ddcuimap.umls import batch_query_pipeline as umls_bqp# from ddcuimap.metamap import batch_query_pipeline as mm_bqp# from ddcuimap.semantic_search import batch_hybrid_query_pipeline as ss_bqpfromddcuimap.hydra_searchimportbatch_hydra_query_pipelineashs_bqpfromddcuimap.utilsimporthelperfromomegaconfimportOmegaConfLOAD/EDIT CONFIGURATION FILEScfg_hydra=helper.compose_config(overrides=["custom=hydra_base"])# cfg_umls = helper.compose_config(overrides=["custom=de", "apis=config_umls_api"])cfg_mm=helper.compose_config(overrides=["custom=de","apis=config_metamap_api"])cfg_ss=helper.compose_config(overrides=["custom=title_def","semantic_search=embeddings","apis=config_pinecone_api",])# # UMLS API CREDENTIALS# cfg_umls.apis.umls.user_info.apiKey = ''# cfg_umls.apis.umls.user_info.email = ''# # MetaMap API CREDENTIALS# cfg_mm.apis.metamap.user_info.apiKey = ''# cfg_mm.apis.metamap.user_info.email = ''## # Pinecone API CREDENTIALS# cfg_ss.apis.pinecone.index_info.apiKey = ''# cfg_ss.apis.pinecone.index_info.environment = ''print(OmegaConf.to_yaml(cfg_hydra))RUN BATCH QUERY PIPELINE# df_umls, cfg_umls = umls_bqp.run_umls_batch(cfg_umls)# df_mm, cfg_mm = mm_bqp.run_mm_batch(cfg_mm)# df_ss, cfg_ss = ss_bqp.run_hybrid_ss_batch(cfg_ss)df_hydra,cfg_step1=hs_bqp.run_hydra_batch(cfg_hydra,cfg_umls=None,cfg_mm=cfg_mm,cfg_ss=cfg_ss)print(df_hydra.head())STEP-1B: *MANUAL CURATION STEP IN EXCELCURATION/SELECTION*see curation example innotebooks/examples_files/DE_Step-1_curation_keepCol.xlsxSTEP-2A: CREATE DATA DICTIONARY IMPORT FILEIMPORT CURATION MODULESfromddcuimap.curationimportcreate_dictionary_import_filefromddcuimap.curationimportcheck_cuisfromddcuimap.utilsimporthelperCREATE DATA DICTIONARY IMPORT FILEcfg_step1=helper.load_config(helper.choose_file("Load config file from Step 1"))df_dd=create_dictionary_import_file.create_dd_file(cfg_step1)print(df_dd.head())STEP-2B: CHECK CUIS IN DATA DICTIONARY IMPORT FILECHECK CUIScfg_step2=helper.load_config(helper.choose_file("Load config file from Step 2"))df_check=check_cuis.check_cuis(cfg_step2)print(df_check.head())Output: Data Dictionary + CUIsBelow is a sample modified data dictionary with curated CUIs after:Running Steps 1-2 ontitlethen taking the generated output dictionary file and;Running Steps 1-2 again onpermissible value descriptionsto get the final output dictionary file.variable nametitledata element concept identifiersdata element concept namesdata element terminology sourcespermissible valuespermissible value descriptionspermissible value output codespermissible value concept identifierspermissible value concept namespermissible value terminology sourcesAgeYrsAge in yearsC1510829;C0001779Age-Years;AgeUMLS;UMLSCaseContrlIndCase control indicatorC0007328Case-Control StudiesUMLSCase;Control;UnknownCase;Control;Unknown1;2;999C1706256;C4553389;C0439673Clinical Study Case;Study Control;UnknownUMLS;UMLS;UMLSSemantic Search with SentenceTransformers Batch QueriesMore documentation to come... Basic pipeline is described below:Subset/Embed/Upsert UMLS Metathesaurus for Pinecone vector databaseStep 1: Subset local copy of UMLS MetathesaurusStep 2: Embed UMLS CUI names and definitions and format metadataStep 3: Upsert embeddings and metadata into Pinecone indexQuery UMLS Metathesaurus vector database with data dictionary embeddingsStep 1: Embed data dictionary fieldsStep 2: Batch Query data dictionary against CUI names and definitions in Pinecone indexStep 3: Evaluate/Curate ResultsStep 4: Create data dictionary based on curationAcknowledgementsThe MetaMap API code included is from Will J Roger's repository -->https://github.com/lhncbc/skr_web_python_apiSpecial thanks to Olga Vovk, Henry Ogoe, and Sofia Syed for their guidance, feedback, and testing of this package.LicenseMIT