Package: aplusml
APLUS MLAPythonLibrary forUsefulnessSimulations ofMachineLearning ModelsCorresponding paper:APLUS - Journal of Biomedical InformaticsCitation:@article{wornow2023aplus,
  title={APLUS: A Python Library for Usefulness Simulations of Machine Learning Models in Healthcare},
  author={Wornow, Michael and Ross, Elsie Gyang and Callahan, Alison and Shah, Nigam H},
  journal={Journal of Biomedical Informatics},
  pages={104319},
  year={2023},
  publisher={Elsevier}
}InstallationRun the following commands to installAPLUS ML:pipinstall-ihttps://test.pypi.org/simple/aplusmlInstallgraphvizbydownloading it here. If you're on Mac withhomebrew, simply run:brew install graphvizMotivationAPLUS ML is a simulation framework for conducting usefulness assessments of machine learning models in workflows.It aims to quantitatively answer the question:If I use this ML model within this workflow, will the benefits outweigh the costs, and by how much?APLUS was originally developed for clinical workflows in healthcare settings, thus all of our examples are healthcare workflow.s. However, APLUS ML is a broadly applicable library to any workflow that involves a machine learning model making decisions on a stream of datapoints, and we encourage contributors from any domain to use and extend APLUS ML.TutorialsWe showcase APLUS on two clinical workflows:Early detection of peripheral artery disease (PAD)Triaging patients for advanced care planning (ACP) consultsJupyter notebooks for these use cases can be found in thetutorials/folder.Early Detection of PADThe code used to generate the figures in our paper is located in thetutorials/directory inpad.ipynb. This notebook loads de-identified patient data from Stanford Hospital, which can be provided upon request.The workflows analyzed can be found in theworkflows/folder. The doctor-driven workflow is inpad_doctor.yamlwhile the nurse-driven workflow is inpad_nurse.yamlThistutorials/pad.ipynbwas used to generate the following figures from the APLUS paper:Triaging Patients for ACP ConsultsThe code used to replicate the findings ofJung et al. 2021can be found in thetutorials/directory inacp_jung_replication.ipynb. This notebook loads de-identified patient data from Stanford Hospital, which can be provided upon request.The workflows analyzed can be found in theworkflows/folder inacp_jung_replication.yamlPlot GallerySome additional example plots that can be generated by APLUS are included below:Conceptual OverviewSimulationWe use discrete event simulation to simulate our workflow $W$.In other words, we represent the world as occuring through a set of discrete, evenly spaced timesteps $\lambda = 0, 1,...,N$. Each timestep $\lambda$ could represent a second, minute, hour, day, etc., the interpretation is up to the user.Events $A$ and $B$ that occur within the same timestep $\lambda$ can have arbitrary ordering if there does not exist a strict $A \rightarrow B$ or $B \rightarrow A$ dependency between these events. In other words, if 3 patients have an MRI and 2 patients have a blood test on timestep $\lambda = 3$, then assuming none of these events are dependent on each other, the ordering in which the blood tests and MRIs occur will be random.A"duration"refers to a number of timesteps (i.e. a length of time).WorkflowA workflow $W$ is simply a set of states $S$.StatesEach state $s \in S$ has associated with it:A duration $\lambda_s$ representing how many timesteps an agent will wait in this state before transitioning to another stateA set of utilities $U_s$A set of resource deltas $R_s \subseteq R$ that specify how various resources $r \in R_s$ change when an agent arrives at this stateA set of transitions $T_s \subseteq T$A type $\tau_s \in {\text{start}, \text{normal}, \text{end}}$.Invariants:$|{ s \in S | \tau_s = \text{start} }| = 1$$ \forall s \in S$ such that $\tau_s = \text{start}, |T_s| > 0$$ \forall s \in S$ such that $\tau_s = \text{normal}, |T_s| > 0$$ \forall s \in S$ such that $\tau_s = \text{end}, |T_s| = 0$TransitionsGiven the set of all transitions $T$, each transition $t \in T_s \subseteq T$ has associated with it:A source state $s \in S$A destination state $s' \in S$ (where $s'$ could be the same as $s$)A duration $\lambda_t$ representing how many timesteps an agent will wait, after having chosen this transition $t$, before moving to state $s'$A condition $c_t \in C$ that, only when TRUE, allows the agent to take this transition $t$ to state $s'$A set of utilities $U_t$A set of resource deltas $R_t \subseteq R$ that specify how various resources $r \in R_t$ change after an agent takes this transitionUtilitiesGiven the set of all utilities $U$, each utility $u \in U$ has associated with it:A value $u_v \in \mathbb{R}$ representing the numeric value of this utilityA unit $u_u$ (i.e. QALYs, US dollars, years, etc.)A condition $c_u \in C$ that, only when TRUE, has the simulation record that this utility value $u_v$ for unit $u_u$ was achievedConditionsA condition $c \in C$ determines whether a utility or transition can be taken. A condition $c$ can take the form of either:A probability (in which case ${ c \in \mathbb{R} | 0 \le c \le 1 }$); ORAn arbitrary Python expression which evaluates to TRUE or FALSEResourcesA resource $r \in R$ is a constrained resource that is shared across all patients. This represents a hospital-level constraint of a workflow (i.e. fiscal budget, number of nurses, MRI machine availability, etc.). Each resource $r$ has associated with it:A level $r_l \in \mathbb{N}$ which represents the current value of the resourceAn initial amount $r_i \in \mathbb{N}$ which ensures $r_l = r_i$ when $\lambda = 0$An maximum capacity $r_m \in \mathbb{N}$ which ensures that $r_l \le r_m$A refill amount $r_a \in \mathbb{N}$ that represents how much this resource gets increased after $\lambda_r$ timesteps have elapsed since the last refillA refill duration $\lambda_r \in \mathbb{N}$ that represents how many timesteps must elapse before the resource is increased to a value of $\max{r_l + r_a, r_m}$!! Important Note !!In order to decrement a resource, you need to specify aresource deltaon the relevant state/transition. Otherwise, if you just require thatnurse_capacity > 0for a transition, then the simulation will not automatically decrementnurse_capacityby 1 when that transition is taken (which can be surprising to some users). This is often a cause of infinite loops, or situations where changing the $r_i$, $r_m$, or $r_a$ of a resource has no effect on the model's achieved utility.PatientsEach patient $p \in P$ has associated with it:A start timestep $\lambda_p$ representing the timestep of the simulation at which the patient began progressing through the workflow (i.e. the day that the patient was admitted to the hospital)A current state $s_p \in S$. The patient always starts at a state $s_p$ where $\tau_{s_p} = \text{start}$A set ofproperties$\Rho_p$ which can be anything (integers, floats, strings, dictionaries, lists, etc.)Ahistoryobject $H_p$ which captures all of the past states, transitions, and utilities that the patient achieved.Running a SimulationEach patient $p$ starts his/her workflow at the state $s$, where $\tau_s = \text{start}$. Note: This is the same for all patientsEach patient $p$ starts his/her workflow at timestep $\lambda_p$. Note: This varies across all patients.Each patient $p$ then progresses through the states of the workflow, according to the applicable transitions and conditions. The patient stops their journey when either of the following conditions are met:The patient reaches a state $s$ where $\tau_s = \text{end}$; ORThe simulation is terminated prematurely after a set number of timesteps have occuredWorkflow SchemaAPLUS requires you to specify the workflow that you want to simulate within a YAML file.The schema of this YAML configuration file is as follows:(+) = optional
{a|b} = must be either string a or b

metadata:
  name (+): str
  path_to_functions (+): str
    => Path to PY file containing Python functions listed in 'variables'
  path_to_properties (+): str
    => Path to CSV file containing Patient properties listed in 'variables'
  properties_col_for_patient_id (+): str
    => Name of column in properties file corresponding to the Patient ID  
  patient_sort_preference_property (+): 
    variable: str
      => Name of property (must be listed in 'variables') to order patients by
    is_ascending: bool
      => If TRUE, then ascending; else descending

variables: dict
  [key]: str
      => Represents ID of state
      => Must be unique
    type: str{scalar|resource|property_dist|property_file|simulation|function} (default = "scalar")
      => 'scalar' = a constant
      => 'resource' = a hospital resource that fluctuates
      => 'property' = a per patient property (from a file or randomly sampled)
      => 'simulation' = tracked by simulation str{sim_current_timestep|time_left_in_sim|time_already_in_sim}
    % If type == 'scalar'...
      value: (int|float|bool|str|list|dict|set)
        NOTE: If specifify a 'set', then need to prepend the set with the '!!set' tag
        NOTE: 'tuple' is not currently supported
    % If type == 'resource'...
      init_amount: int
      max_amount: int
      refill_amount: int
      refill_duration: int
    % If type == 'property'...
      % Either load from file...
        column: str
      % or constant...
        value: Any
      % or randomly sample...
        distribution: str{bernoulli|exponential|binomial|normal|poisson|uniform}
        mean (+): float
        std (+): float
        start (+): float
        end (+): float

states: dict
  [key]: str
      => Represents ID of state
      => Must be unique
    label (+): str (default = value of 'id')
    type (+): str{start|end|intermediate} (default = "intermediate")
    duration (+): float (default = 0.0)
      => Waiting this number of timesteps occurs AS SOON AS this state is hit (so BEFORE any transitions from it are calculated)
    utilities (+): str|float|bool|list[dict] (default = 0.0)
      => If not a list[dict], then the expression specified as evaluated as Python
      - value (+): float|str (default = 0.0)
          => If str, then assume it's a function
        if (+): str
          => String is a conditional expression
          => If TRUE, then set 'value' as utility for this 'unit'
          => NOTE - These 'if' statements are not mutually exclusive (i.e. multiple ones will simply be summed together)
        unit (+): str (default = "")
    resource_deltas (+): dict[float] (default = {})
      => [key] = resource from 'variables', [value] = how much to change each resource level AS SOON AS this state is hit (so BEFORE any transitions from it are calculated, but AFTER the duration has occurred)
    TODO: property_updates (+): dict[float] (default = {})
      => [key] = property from 'variables', [value] = new value of this property for patient AS SOON AS this state is hit
    transitions (+): list[dict] (default = [])
        dest: str
          => Must match ID of a state
        label (+): str (default = "")
        % Can either have...
        % - All transitions have an 'if' condition (where if the last transition doesn't have an 'if', it defaults to always TRUE)
        % - All transitions have a 'prob' condition (where if the last transition doesn't have a 'prob', it defaults to = 1 - (sum of other probs))
        % - The first half of transitions have an 'if' condition, but the second have of transitions have a 'prob' (all 'if' transitions must have an 'if', the 'prob' are evaluated conditional on all 'if' being FALSE, and the last 'prob' transition defaults to = 1 - sum(other probs))
          if (+): bool|str (default = bool:true)
            => String is a conditional expression
            => Must come BEFORE 'prob' (if mixed)
            => Must always be at least one TRUE condition across all transitions for this state (unless mixed with 'prob')
            => Conditionals will be evaluted in order and break on first TRUE
            => If last 'if' isn't specified, defaults to TRUE
          prob (+): float|str (default = 1 - (sum of other probs))
            => String is a variable
            => Must come AFTER 'if' (if mixed)
              => If mixed, then 'prob' is conditional probability given all 'if' are FALSE
            => Must sum to 1 across all 'prob' transitions for this state
            => If last 'prob' isn't specified, defaults to = 1 - (sum of other probs)
          ================
        %
        duration (+): float (default = 0.0)
          => Waiting this number of timesteps occurs BEFORE this transition is taken
        utilities (+): str|float|bool|list[dict] (default = 0.0)
          => Same as for state
        resource_deltas (+): dict[float] (default = {})
          => [key] = resource from 'variables', [value] = how much to change each resource level by taking this transition (so AFTER any transitions from it are calculated)DevelopmentInstallation# Download repogitclonehttps://github.com/som-shahlab/aplus.gitcdaplus# Create environmentcondacreate-napluspython=3.10-y
condaactivateaplus
pip3install-e.
pip3install-rrequirements.txtRepository File StructureCore APLUS module insrc/aplusml(listed in the order that they should be used):parse.py- Functions to parse YAML files into Python objects for use insim.pysim.py- Core simulation engine which progresses patients through the desired clinical workflowrun.py- Wrapper functions aroundsim.pyto help run/track multiple simulationsplot.py- Plotting functionsmodels.py- Classes for entities like patients, patient history, utilities, resources, etc.draw.py- Functions to draw the workflow graphSupporting files:tutorials/- Contains Jupyter notebooks that demonstrate how to use APLUSpad.ipynb- Demonstrates how to use APLUS to simulate the novel PAD workflow described in the paperpad.py- Helper functions for PAD-specific workflow analysisacp_jung_replication.ipynb- Demonstrates how to use APLUS to replicate the plots ofJung et al. 2021workflows/- Contains YAML files that define the workflows analyzed in the paperpad_doctor.yaml- The doctor-driven PAD workflowpad_nurse.yaml- The nurse-driven PAD workflowacp_jung_replication.yaml- The exact same ACP workflow analyzed inJung et al. 2021tests/- Contains unit tests for the APLUS frameworkrun_tests.py- Script to run all unit teststest_*.py- Tests for each moduletest*.yaml- Workflow YAML files for each corresponding testutils.py- Utility functions for testinginput/- Contains input data fed into the simulationoutput/- Contains output data from the simulations (this is useful for caching results so you don't have to re-run time-consuming simulations)TestsThe filetests/run_tests.pyruns all of thetest[d].pyfiles in thetests/directory. Eachtest[d].pyfile has a correspondingtest[d].yamlfile that serves as its input.To run tests:cd tests
python3 run_tests.py