Package: xcube
#|hide
#| eval: false
! [ -e /content ] && pip install -q condacolab && python3 -c  "import condacolab; condacolab.install()" # create conda environment in colab
! [ -e /content ] && pip install -Uqq xcube  # upgrade xcube on colab#| hide
from xcube.text.all import *

%load_ext autoreload
%autoreload 2xcubexcube trains and explains XMTC modelsEXplainable EXtreme Multi-Label TeXt Classification:What is XMTC?Extreme Multi-Label Text Classification (XMTC) addresses the problem of automatically assigning each data point with most relevant subset of labels from an extremely large label set. One major application of XMTC is in the global healthcare system, specifically in the context of the International Classification of Diseases (ICD). ICD coding is the process of assigning codes representing diagnoses and procedures performed during a patient visit using clinical notes documented by health professionals.Datasets?Examples of ICD coding dataset:MIMIC-IIIandMIMIC-IV. Please note that you need to be a credentialated user and complete a training to acces the data.What is xcube?xcube trains and explains XMTC models using LLM fine-tuning.InstallCreate new conda environment:condacreate-nxxxpython=3.10condaactivatexxxInstall PyTorch with cuda enabled: [Optional]condasearchpytorchuse the build string that matches the python and cuda version, replacing the pytorch version and build string appropriately:condainstallpytorch=2.0.0=cuda118py310h072bc4cpytorch-cuda=11.8-cpytorch-cnvidiaUpdate cuda-toolkit:sudoaptinstallnvidia-cuda-toolkitVerify cuda is available:  Runpythonandimport torch; torch.cuda.is_available()Install using:pipinstallxcubeConfigure accelerate by:accelerateconfigHow to useYou can either clone the repo and open it in your own machine. Or if you don't want to setup a python development environment, an even easier and quicker approach is to open this repo usingGoogle Colab. You can open this readme page in Colab using thislink.IN_COLAB = is_colab()source_mimic3 = untar_xxx(XURLs.MIMIC3_DEMO)
source_mimic4 = untar_xxx(XURLs.MIMIC4)
path = Path.cwd().parent/f"{'xcube' if IN_COLAB else ''}" # root of the repo
(path/'tmp/models').mkdir(exist_ok=True, parents=True)
tmp = path/'tmp'Check your GPU memory! If you are running this on google colab be sure to turn on the GPU runtime. You should be able to train and infer all the models with atleast 16GB of memory. However, note that training the full versions of the datasets from scratch requires atleast 48GB memory.cudamem()Train and Infer on MIMIC3-rare50MIMIC3-rare50 refers to a split ofMIMIC-IIIthat contains the 50 most rare codes (Refer toKnowledge Injected Prompt Based Fine-Tuning for Multi-label Few-shot ICD Codingfor split creation).data = join_path_file('mimic3-9k_rare50', source_mimic3, ext='.csv')
!head -n 1 {data}df = pd.read_csv(data,
                 header=0,
                 names=['subject_id', 'hadm_id', 'text', 'labels', 'length', 'is_valid', 'split'],
                 dtype={'subject_id': str, 'hadm_id': str, 'text': str, 'labels': str, 'length': np.int64, 'is_valid': bool, 'split': str})
df.head(2)To launch the training of an XMTC model on MIMIC3-rare50:os.chdir(path/'scripts')
!./run_scripts.sh --script_list_file script_list_mimic3_rare50trainTrain and Infer on MIMIC3-top50MIMIC3-top50 refers to a split ofMIMIC-IIIthat contains 50 most frequent codes (Refer toExplainable Prediction of Medical Codes from Clinical Textfor split creation)data = join_path_file('mimic3-9k_top50', source_mimic3, ext='.csv')
!head -n 1 {data}df = pd.read_csv(data,
                 header=0,
                 names=['subject_id', 'hadm_id', 'text', 'labels', 'length', 'is_valid', 'split'],
                 dtype={'subject_id': str, 'hadm_id': str, 'text': str, 'labels': str, 'length': np.int64, 'is_valid': bool, 'split': str})
df.head(2)To infer one our pretrained XMTC models on MIMIC3-top50 (Metrics for inference - Precision@3,5,8,15):model_fnames = L(source_mimic3.glob("**/*top50*.pth")).map(str)
print('\n'.join(model_fnames))
fname = Path(shutil.copy(model_fnames[2], tmp/'models')).name.split('.')[0]
print(f"We are going to infer model {fname}.")os.chdir(path/'scripts')
!./launches/launch_top50_mimic3 --fname {fname} --no_running_decoder --infer 1To launch the training of an XMTC model on MIMIC3-top50 from scratch:#| eval: false
os.chdir(path/'scripts')
!./run_scripts.sh --script_list_file script_list_mimic3_top50trainTrain and Infer on MIMIC3-full:MIMIC3-full refers to the fullMIMIC-IIIdataset. (Refer toExplainable Prediction of Medical Codes from Clinical Textfor details of how the data was curated)data = join_path_file('mimic3-9k_full', source_mimic3, ext='.csv')
!head -n 1 {data}df = pd.read_csv(data,
                 header=0,
                 names=['subject_id', 'hadm_id', 'text', 'labels', 'length', 'is_valid', 'split'],
                 dtype={'subject_id': str, 'hadm_id': str, 'text': str, 'labels': str, 'length': np.int64, 'is_valid': bool, 'split': str})
df.head(2)Lets's look at some of the ICD9 codes description:des = load_pickle(source_mimic3/'code_desc.pkl')
lbl_dict = dict()
for lbl in df.labels[1].split(';'):
    lbl_dict[lbl] = des.get(lbl, 'NF')
pd.DataFrame(lbl_dict.items(), columns=['icd9_code', 'desccription'])To infer one our pretrained XMTC models on MIMIC3-full (Metrics for inference - Precision@3,5,8,15):model_fnames = L(source_mimic3.glob("**/*full*.pth")).map(str)
print('\n'.join(model_fnames))
fname = Path(shutil.copy(model_fnames[0], tmp/'models')).name.split('.')[0]
print(f"Let's infer the pretrained model {fname}.")os.chdir(path/'scripts')
!./launches/launch_complete_mimic3 --fname {fname} --infer 1 --no_running_decoderTrain and Infer on MIMIC4-full:MIMIC4-full refers to the fullMIMIC-IVdataset using ICD10 codes. (Refer toAutomated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Studyfor details of how the data was curated)data = join_path_file('mimic4_icd10_full', source_mimic4, ext='.csv')
!head -n 1 {data}df = pd.read_csv(data,
                    header=0,
                    usecols=['subject_id', '_id', 'text', 'labels', 'num_targets', 'is_valid', 'split'],
                    dtype={'subject_id': str, '_id': str, 'text': str, 'labels': str, 'num_targets': np.int64, 'is_valid': bool, 'split': str})
df.head(2)Let's look at some of the descriptions of ICD10 codes:stripped_codes = [''.join(filter(str.isalnum, s)) for s in df.labels[0].split(';')]
desc = get_description(stripped_codes)
pd.DataFrame(desc.items(), columns=['icd10_code', 'desccription'])To infer one our pretrained XMTC models on MIMIC4-full (Metrics for inference - Precision@5,8,15):print('\n'.join(L(source_mimic4.glob("**/*full*.pth")).map(str)))
model_fname = Path('/home/deb/.xcube/data/mimic4/mimic4_icd10_clas_full.pth')
fname = Path(shutil.copy(model_fname, tmp/'models')).name.split('.')[0]
print(f"Let's infer the pretrained model {fname}.")#| eval: false
os.chdir(path/'scripts')
!./launches/launch_complete_mimic4_icd10 --fname mimic4_icd10_clas_full --no_running_decoder --infer 1AcknowledgementThis repository is my attempt to create Extreme Multi-Label Text Classifiers using Language Model Fine-Tuning as proposed byJeremy HowardandSebastian RuderinULMFit. I am also heavily influenced by thefast.ai'scoursePractical Deep Learning for Codersand the excellent libraryfastai. I have adopted the style of coding fromfastaiusing the jupyter based dev environmentnbdev. Since this is one of my fast attempt to create a full fledged python library, I have at times replicated implementations from fastai with some modifications. A big thanks to Jeremy and his team fromfast.aifor everything they have been doing to make AI accessible to everyone.