Package: bloatectomy
BloatectomyBloatectomy: a method for the identification and removal of duplicate text in the bloated notes of electronic health records and other documents. Takes in a list of notes or a single file (.docx, .txt, .rtf, etc) or single string to be marked for duplicates. Marked output and tokens are output.RequirementsPython>=3.7.x (in order for the regular expressions to work correctly)resyspandas (optional, only necessary if using MIMIC III data)docx (optional, only necessary if input or output is a word/docx file)Installationusing anaconda or minicondaconda install -c summerkrankin bloatectomyusing pip via PyPImake sure to install it to python3 if your default is python2python3 -m pip install bloatectomyusing pip via githubpython3 -m pip install git+git://github.com/MIT-LCP/bloatectomymanual install by cloning the repositorygit clone git://github.com/MIT-LCP/bloatectomy
cd bloatectomy
python3 setup.py installExamplesTo run bloatectomy on a sample string with the following options:highlighting duplicatesdisplay raw resultsoutput file as htmloutput file of numbered tokens:from bloatectomy import bloatectomy

text = '''Assessment and Plan
61 yo male Hep C cirrhosis
Abd pain:
-other labs: PT / PTT / INR:16.6//    1.5, CK / CKMB /
ICU Care
-other labs: PT / PTT / INR:16.6//    1.5, CK / CKMB /
Assessment and Plan
'''

bloatectomy(text, style='highlight', display=True, filename='sample_txt_highlight_output', output='html', output_numbered_tokens=True)To use with example text or load ipynb examples, download the repository or just the bloatectomy_examples foldercd bloatectomy_examples
from bloatectomy import bloatectomy

bloatectomy('./input/sample_text.txt',
            style='highlight', display=False,
            filename='./output/sample_txt_highlight_output',
            output='html',
            output_numbered_tokens=True,
            output_original_tokens=True)DocumentationThe paper is located at TBAclass bloatectomy(input_text,
                  path = '',
                  filename='bloatectomized_file',
                  display=False,
                  style='highlight',
                  output='html',
                  output_numbered_tokens=False,
                  output_original_tokens=False,
                  regex1=r"(.+?\.[\s\n]+)",
                  regex2=r"(?=\n\s*[A-Z1-9#-]+.*)",
                  postgres_engine=None,
                  postgres_table=None)Parametersinput_text: file, str, listAn input document (.txt, .rtf, .docx), a string of text, or list of hadm_ids for postgres mimiciii database or the raw text.style: str, optional, default=highlightMethod for denoting a duplicate. The following are allowed:highlight,bold,remov.filename: str, optional, default=bloatectomized_fileA string to name output file of the bloat-ectomized document.path: str, optional, default=' 'The directory for output files.output_numbered_tokens: bool, optional, default=FalseIf set toTrue, a .txt file with each token enumerated and marked for duplication, is output as[filename]_token_numbers.txt. This is useful when diagnosing your own regular expression for tokenization or testing theremovoption forstyle.output_original_tokens: bool, optional, default=FalseIf set toTrue, a .txt file with each original (non-marked) token enumerated but not marked for duplication, is output as[filename]_original_token_numbers.txt.display: bool, optional, default=FalseIf set toTrue, the bloatectomized text will display in the console on completion.regex1: str, optional, default=r"(.+?\.[\s\n]+)"The regular expression for the first tokenization. Split on a period (.) followed by one or more white space characters (space, tab, line breaks) or a line feed character (\n). This can be replaced with any valid regular expression to change the way tokens are created.regex2: str, optional, default=r"(?=\n\s*[A-Z1-9#-]+.*)"The regular expression for the second tokenization. Split on any newline character (\n) followed by an uppercase letter, a number, or a dash. This can be replaced with any valid regular expression to change how sub-tokens are created.postgres_engine: str, optional
The postgres connection. Only relevant for use with the MIMIC III dataset. When data is pulled from postgres the hadm_id of the file will be appended to thefilenameif set or the defaultbloatectomized_file. See the jupyter notebookmimic_bloatectomy_examplefor the example code.postgres_table: str, optional
The name of the postgres table containing the concatenated notes. Only relevant for use with the MIMIC III dataset. When data is pulled from postgres the hadm_id of the file will be appended to thefilenameif set or the defaultbloatectomized_file. See the jupyter notebookmimic_bloatectomy_examplefor the example code.