Package: report-guided-annotation
Report Guided AnnotationBackgroundDeep learning-based diagnostic performance increases with more annotated data, but manual annotation is a bottleneck in most fields. Experts evaluate diagnostic images during clinical routine, and write their findings in reports. Automatic annotation based on clinical reports could overcome the manual labelling bottleneck. This repository implements the generation of dense annotations from model predictions, guided by the target number of structures.📖 CitationIf you use this algorithm in your work, please cite our work:J. S. Bosma, A. Saha, M. Hosseinzadeh, I. Slootweg, M. de Rooij, and H. Huisman (2022), "Semi-supervised learning with report-guided lesion annotation for deep learning-based prostate cancer detection in bpMRI", to be submittedInstallationReport Guided Annotation is pip-installable:pip install report_guided_annotationUsagePreparationTo generate annotations from softmax predictions, prepare the input folder in the following format:path/to/softmax/folder/
├── num_lesions_to_retain_map.json
├── ProstateX-0000.nii.gz
├── ProstateX-0001.nii.gz
├── ProstateX-0002.nii.gz
...Seetests/inputfor an example. Allowed file extensions:.nii.gz,.nii,.mha,.mhd,.npy, and.npzfrom nnUNet.The number of lesions to retain should be stored as a json dictionary, with the filenames of the predictions as keys, and the number of structures to retain as values (in this example, the PI-RADS ≥ 3 lesions were counted):{"ProstateX-0000.nii.gz":1,"ProstateX-0001.nii.gz":1,"ProstateX-0002.nii.gz":2,...}GenerationGenerate automatic annotations from the command line:python3 -m report_guided_annotation --input=/path/to/input/folder/ --output=/path/to/output/folderOr from a Python script:fromreport_guided_annotationimportcreate_automatic_annotations_for_foldercreate_automatic_annotations_for_folder(input_dir="/path/to/input/folder/",output_dir="/path/to/output/folder/")MechanismFrom each softmax volume we create distinct lesion candidates, as illustrated in the figure below. Specifically, we create a lesion candidate by starting at the most confident voxel, and including all connected voxels (in 3D) with at least 40% of the peak's confidence. After a lesion candidate is extracted, we remove it from the model prediction and continue to the next peak.The automatic voxel-level annotations are generated by keeping the nsigmost confident lesion candidates, with nsigthe number of clinically significant report findings (in this example, the number of PI-RADS ≥ 3 lesions). If there are fewer lesion candidates than clinically significant report findings, the case is excluded (no automatic annotation will be saved, unless explicitly requested with--skip_if_insufficient_lesions=0orskip_if_insufficient_lesions=False).The masks for each finding is binarized and saved to the output folder.Lesion candidate extraction from softmax prediction:This is how report-guided automatic annotations would fit in an end-to-end pipeline:Extract PI-RADS Scores from Radiology ReportThe PI-RADS score extraction from radiology report was designed with the reports from RUMC in mind. Please verify if your institution's reports are similar, and modify the code accordingly!Input:Radiology report for prostate cancer detection, reported with PI-RADS v2.Mechanism:Most of the radiology reports in our dataset were generated from a template, and
modified to provide additional information. Although multiple templates were used
over the years, this resulted in structured reports for most visits. This makes a
rule-based natural language processing script a reliable and transparent way to
extract PI-RADS scores from our radiology reports.Simply counting the occurrences ofPI-RADS 4/5in the report body is reasonably
effective, but has some pitfalls. For example, prior PI-RADS scores are often
referenced during follow-up visits, resulting in false positive matches. Findings
can also be grouped and described jointly, resulting in false negatives. To improve
the reliability of the PI-RADS extraction from radiology reports, we extract the
scores in two steps.First, we try to split the radiology reports in sections for individual findings,
by searching for text that matches the following structure:[Finding] (number indicator) [number]WhereFindingmatches the Dutch translationsafwijking,laesie,markeringorregio. The optional number indicators arenr.,markandnummer. The
number at the end matches one or multiple numbers (e.g.,1or2+3).Secondly, we extract the PI-RADS scores by searching for text that matches the
following structure:[PI-RADS] (separators) [number 1-5]Where the optional separators includev2 categoryand:. The dash betweenPIand `RADS' is optional. The T2W, DWI and DCE scores are extracted analogous to the
PI-RADS score, while also allowing joint extraction:T2W/DWI/DCE score: [1-5]/[1-5]/[-+]In this instance, the first number is matched with the T2W score, the second with
DWI and the+or-with DCE.In case the report could not be split in sections per lesion, we apply strict pattern
matching on the full report. During strict pattern matching we only extract T2W, DWI
and DCE scores jointly, to ensure the scores are from the same lesion. The resulting
PI-RADS scores are extracted from the full report and matched to the individual scores.Example report sections with the text matching the PI-RADS, T2W, DWI and DCE scores
coloured in are shown in J. S. Bosma, et. al. "Report Guided Automatic Lesion Annotation
for Deep Learning Prostate Cancer Detection in bpMRI", to be submitted, Figure 2.Output:For each report, the PI-RADS scores are returned in the following format:[(1,{'T2W':'3','DWI':'5','DCE':'+','tot':5}),(2,{'T2W':'2','DWI':'2','DCE':'+','tot':2})]UsageThe scores can be extracted using a Python script:fromreport_guided_annotationimportextract_pirads_scores# read radiology reportwithopen("tests/reports/sample_1.txt")report=fp.read()# extract PI-RADS scores from radiology reportscores=extract_pirads_scores(report=report)To combine the report reading with the report guided annotation procedure, parse the reports and save the result in the format described above:importosimportjsonfromreport_guided_annotationimportextract_pirads_scorespath_to_reports="tests/reports"path_to_softmax="tests/parse_reports_output"num_lesions_to_retain_map={}forfninos.listdir(path_to_reports):ifnot'.txt'infn:print(f"Skipped{fn}, not a report?")# read radiology reportwithopen(os.path.join(path_to_reports,fn))asfp:report=fp.read()# extract PI-RADS scores from radiology reportscores=extract_pirads_scores(report=report)# count number of PI-RADS >= 4 lesionspirads_scores=[int(lesion_scores['tot'])for(lesion_id,lesion_scores)inscoresiflesion_scores['tot']isnotNone]num_pirads_45=sum([score>=4forscoreinpirads_scores])# store number of clinically significant report findingssoftmax_fn=fn.replace(".txt",".nii.gz")num_lesions_to_retain_map[softmax_fn]=num_pirads_45# save number of lesion candidates to retainwithopen(os.path.join(path_to_softmax,"num_lesions_to_retain_map.json"),"w")asfp:json.dump(num_lesions_to_retain_map,fp,indent=4)