<figure><img src="image_1.png" /></figure>

HAL
open science

# Exploring the values underlying machine learning research in medical image analysis

John S.H. Baxter, Roy Eagleson

<figure><img src="image_2.png" /></figure>

## To cite this version:

John S.H. Baxter, Roy Eagleson. Exploring the values underlying machine learning research in medical image analysis. Medical Image Analysis, 2025, 102, pp.103494. 10.1016/j.media.2025.103494. hal-04969693

HAL Id: hal-04969693

https://hal.science/hal-04969693v1

Submitted on 27 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Review

# Exploring the values underlying machine learning research in medical image analysis

John S.H. Baxter<sup>a, ID,*</sup>, Roy Eagleson<sup>b, ID</sup>

ARTICLE INFO

Keywords:

Research values

Machine learning

Intermediate representations

Philosophy of science

ABSTRACT

Machine learning has emerged as a crucial tool for medical image analysis, largely due to recent developments in deep artificial neural networks addressing numerous, diverse clinical problems. As with any conceptual tool, the effective use of machine learning should be predicated on an understanding of its underlying motivations just as much as algorithms or theory — and to do so, we need to explore its philosophical foundations. One of these foundations is the understanding of how values, despite being non-empirical, nevertheless affect scientific research. This article has three goals: to introduce the reader to values in a way that is specific to medical image analysis; to characterise a particular set of technical decisions (what we call the *end-to-end vs. separable learning spectrum*) that are fundamental to machine learning for medical image analysis; and to create a simple and structured method to show how these values can be rigorously connected to these technical decisions. This better understanding of how the philosophy of science can clarify fundamental elements of how medical image analysis research is performed and can be improved.

Contents

<table><tbody><tr><td>1.</td><td>Introduction</td><td>2</td></tr><tr><td>2.</td><td>Research values and technical decisions</td><td>3</td></tr><tr><td>2.1.</td><td>How do values effect technical decisions?</td><td>3</td></tr><tr><td>2.2.</td><td>A hierarchy of values for algorithm research</td><td>4</td></tr><tr><td>2.3.</td><td>Ambiguous value terms</td><td>5</td></tr><tr><td>3.</td><td>Intermediate representations and the end-to-end vs. separable learning spectrum</td><td>5</td></tr><tr><td>3.1.</td><td>Example: Subcortical brain anatomy segmentation</td><td>6</td></tr><tr><td>3.2.</td><td>Example: Cardiac performance measurement</td><td>6</td></tr><tr><td>3.3.</td><td>Example: Breast cancer screening</td><td>7</td></tr><tr><td>4.</td><td>Research values underlying the end-to-end vs separable learning spectrum</td><td>7</td></tr><tr><td>4.1.</td><td>The idealised ML research process</td><td>7</td></tr><tr><td>4.2.</td><td>Connecting technical decisions with external values</td><td>8</td></tr><tr><td>4.2.1.</td><td>Problem design and architecture specification</td><td>8</td></tr><tr><td>4.2.2.</td><td>Dataset considerations</td><td>9</td></tr><tr><td>4.2.3.</td><td>Training and evaluation considerations</td><td>10</td></tr><tr><td>4.2.4.</td><td>Knowledge discovery considerations</td><td>10</td></tr><tr><td>4.2.5.</td><td>Clinical translation and workflow considerations</td><td>11</td></tr><tr><td>4.2.6.</td><td>Publication considerations</td><td>11</td></tr><tr><td>4.2.7.</td><td>Real-world use and external values</td><td>11</td></tr><tr><td>4.3.</td><td>Other values in medical imaging and ML</td><td>12</td></tr><tr><td>4.3.1.</td><td>Accuracy, performance, and universality</td><td>12</td></tr><tr><td>4.3.2.</td><td>Automation</td><td>12</td></tr><tr><td>4.3.3.</td><td>Ease-of-implementation</td><td>12</td></tr><tr><td>4.3.4.</td><td>Explainability and interpretability</td><td>12</td></tr></tbody></table>

<footnote><sup>a</sup> Laboratoire Traitement du Signal et de l'Image (LTSI, INSERM UMR 1099), Université de Rennes, Rennes, France</footnote>

<footnote><sup>b</sup> Biomedical Engineering Graduate Program, Western University, London, Canada</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail address: john.baxter@univ-rennes.fr (J.S.H. Baxter).</footnote>
<table><tr><td>4.3.5. Fairness</td><td>12</td></tr><tr><td>4.3.6. Security</td><td>12</td></tr><tr><td>5. On the genealogy of models via related values</td><td>13</td></tr><tr><td>6. Conclusions</td><td>13</td></tr><tr><td>CRediT authorship contribution statement</td><td>14</td></tr><tr><td>Declaration of competing interest</td><td>14</td></tr><tr><td>Data availability</td><td>14</td></tr><tr><td>References</td><td>14</td></tr></table>

# 1. Introduction

Despite the common perception of scientific research as entirely objective, philosophers of science have long agreed that it actually heavily depends on non-empirical and often subjective considerations, i.e. research values. Several of these values are undoubtedly relevant to machine learning (ML) in medical image analysis, such as:

* high accuracy and robustness;

* fairness and freedom from bias;

* explainability and interpretability;

* data security and privacy; and

* minimal environmental impact.

Some of these values can be rendered objective through one or more well-defined qualitative metrics, especially if there is a high degree of consensus in the community surrounding these metrics. For example, the objectified forms of *high accuracy* as a value for image segmentation, object detection, and classification has largely come under a consensus through frameworks such as the Metrics flowchart (Reinke et al., 2022). For other values such as fairness, there are several metrics available (Mienye et al. (2024), for example, describe six different groups of fairness metrics, which do not fully overlap with the six groups identified by Caton and Haas (2024), illustrating both variety and diversity within just the domain of classification problems in healthcare) but there is lively discussion in the community as to how well different metrics fit different contexts. Other values, such as explainability and interpretability, are even further from objectivity in that they do not yet have a series of widely agreed-upon metrics; that different researchers often simply state them as non-empirical properties of the system. Regardless of the degree to which these values can be objectively measured, their more subjective definitions have an impact on how research is prioritised and performed especially once identified and better understood.

Indeed, the emergence of any conceptual tool (any mathematical, theoretical, or practical tool or standard that facilitates research, such as an algorithm, piece of equipment, definition of a specific term, etc...) brings with it some particular values that are implicit in its design and thereby influence its use. This includes conceptual tools widely accepted by their communities to the point of unanimity, such as randomised controlled clinical trials (Edwards et al., 1998; Bluhm, 2017; Lalumera and Fanti, 2019) or the translation of research into personalised medicine (Gefenas et al., 2011; Savard, 2013). But it also includes less obvious ones, such as the specifics of how to understand and formulate the approach to a particularly complex medical image analysis problem. Modern ML, specifically deep learning, as a conceptual tool has unquestionably changed much of medical image analysis research. The proliferation of deep neural networks has allowed numerous disparate problems to be addressed with readily available technological tools in a single coherent framework (i.e. gradient-descent based learning of large artificial neural networks). In some cases, the same architectures can even be used without any modification for different problems with a high degree of performance (Isensee et al., 2018). Thus, it becomes necessary to understand what systems of values underlies the use of deep learning and influence specific decisions made within this medical image analysis paradigm.

Some have already noted the ethical impact of medical ML, especially in terms of the introduction of hidden intrinsic biases (Grote, 2021; Baxter and Jannin, 2022a; Eagleson, 2024), carbon intensiveness (Jannin, 2021), and data/economic colonialism (Souza et al., 2023). Disagreements in terms of which values are espoused and to what degree should not detract from discussions in medical ML research. Instead, these variations should add to critical analyses about how particular research should be performed or about the types of ML solutions that should be constructed in said research, provided that the researchers involved understand the causes and effects of these values in relation to choices made during the research process. This, however, is a non-trivial task as there is a significant number of interrelated technical decisions in any research project and each of these can be motivated (implicitly or explicitly) in different ways by different researchers. Often, certain technical decisions are coupled together by their motivations, with others being more-or-less distinct. The goal of this article is to shed light on this situation for a particular family of technical decisions made in any ML project for medical image analysis.

## Contributions

This article, an extension of our previous conference paper (Baxter, 2022), has three separate but related goals corresponding to its first three sections.

Section 2 more rigorously defines research values from the philosophy of science and then examines what it would mean for a value to motivate a technical decision. We observe a separation between values as they are generally understood and their influence on specific technical choices. We propose that this can be bridged by considering how values engender other values, ordering them by their immediacy to technical decisions. Values on the lower end are not universally held but uncontroversially lead to a particular technical decision. Values on the higher end are the opposite: they are more widely held but are more ambiguous in terms of how they relate to specific technical decisions.

Section 3 explores a particular set of technical decisions that define the end-to-end vs. separable learning spectrum. This spectrum groups together different approaches in terms of how much they use explicit intermediate representations which frame a computation as the act of transforming data structures or objects into each other. The idea of this spectrum is that some models (i.e. end-to-end ones) try to minimise these representations whereas others (i.e. separable ones) take advantage of them.

Section 4 uses approach outlined in Section 2 to explore the values associated with the spectrum from Section 3 in a way that mirrors the idealised ML research pipeline. This leads to the identification of fifteen specific values and three dichotomies related to technical decisions at the heart of ML in medical image analysis and encourage a particular end of the spectrum. For completeness, we also discuss several values that seem connected to a particular end of the spectrum but are not entirely so.

Sections 5 and 6 conclude with some hypotheses and discussion about how research values could have historically arisen in terms of machine learning as well as general remarks about the applicability of research values to understanding medical image analysis.

This paper tries to speak to two largely disparate audiences. For the philosophy of science community, who are already familiar with the
notion of research values, we intend to provide an introduction to the technical nuances of ML in medical imaging research that they may not have previously encountered. For members of the medical image analysis community, we hope to show another perspective for organising their technical knowledge into philosophically-relevant structures, allowing for deeper and more critical analysis. For both, we find it important to also illustrate how research values permeate the entirety of the technical research process. An exhaustive treatment of all research values and all technical decisions is not feasible: such an investigation would require several volumes at bare minimum. Thus, we have tried to introduce just enough philosophical and technical concepts to motivate and justify a deep dive into one particular element that we hope both communities will find interesting and that might help guide ML research towards technical implementations that fundamentally align with the rationale in which they were developed.

## 2. Research values and technical decisions

Before exploring how values relate to medical image analysis, it is useful to have a definition of *value* from the philosophy of science. Taking a general definition provided by Elliott (2017), “a value is something that is desirable or worthy of pursuit” (pg. 11). One can easily see numerous values here that intersect with ML in medicine, such as those mentioned in the previous section. It is not uncommon for values to come into play when considering not only research into a particular technology but the potential impact of its use.

Philosophers of science have generally split values into two groups: *epistemic* values (i.e. reproducibility, falsifiability, logical consistency, explanatory power, etc...) that fundamentally reflect some desirable property to have in the knowledge generated by scientific research; and *non-epistemic* values (i.e. fairness, economic potential, ecological considerations, etc...) which focus on the impact of the research process or resulting product on larger systems such as society, the environment or the economy. The characteristic (albeit contentious) difference between epistemic and non-epistemic values is that the former relate to qualities specific only to knowledge and understanding, such as the ease at which an experiment can be reproduced or replicated or the clarity with which a theory explains the mechanism behind a particular phenomenon. The latter would cover everything else, but it is largely constrained to values that are cultural, social, or otherwise dependent on the context in which the knowledge or understanding takes place, rather than the knowledge or understanding itself (Rooney, 1992).

Thus, epistemic values are often seen as more removed from the ethical concerns that most people, included philosophers of science, think of when discussing the role of values in science. However, these values are more widely discussed among the researchers themselves (Birhane et al., 2022). Certain epistemic values are seen as central to the nature of science, notably reproducibility and falsifiability (Popper, 1963), with some philosophers arguing that they should be the only ones relevant to how science is performed (Betz, 2017). His key argument is that value-laden decisions should be left to policy-makers and that scientists should play the neutral role of evidence gatherers and interpreters, clarifying where uncertainties and caveats exist.

Non-epistemic values are generally more ethics-laden and highly variable depending on the research domain. Jannin (2021) broadly classifies these values (with respect to digital health technology) as either ecological (i.e. factors that affect the environment), societal (i.e. factors that effect human systems such as economies or political systems), or social (i.e. the accessibility and applicability of technology to particular individuals). Douglas' (2000) divides medical research values among the political, ethical, and social. Although there is some disagreement into how much these values should affect how science is performed, it is unanimously understood that they do affect what hypotheses are chosen to be subjected to the scientific process. That is, scientists choose what research to perform based on what they believe to be important to society.

There has been a persistence among some philosophers of science (notably Betz (2013,2017)) to make science more objective by making it more agnostic to non-epistemic values. Others, such as Douglas (2000), argue that it is impossible to disentangle even clearly non-epistemic values from research in sciences that directly affect humans, such as medicine, because of the problem of *inductive risk*. Inductive risk refers to the potential harms of accepting a false hypothesis or rejecting a true one. This often involves the decision of what *p*-value to consider significant but also what evidence (e.g. experiments, datasets, metrics) is considered appropriate for operationalising a hypothesis. She argues that scientists must make judgments here that are not strictly scientific nor epistemic, but depend on non-epistemic values. For an example in medical image processing, answering a question such as “Should I validate my algorithm on a phantom?” depends on a non-empirical value judgment about whether or not results on a particular phantom constitute relevant evidence for a particular purpose.

Inductive risk is further complicated in medical imaging by a concept introduced by Lalumera and Fanti (2019): the *prevalence of treatment*. Their argument is that, for the highest levels of clinical validation (i.e. rate of survival, complications, or death), the efficacy of medical imaging is overshadowed by the efficacy of the treatment that patients could undergo. That is, for imaging to have a positive impact on the patient's lifespan or quality-of-care, we must have a sufficiently effective treatment for whatever disorder we are trying to detect. The argument from the prevalence of treatment suggests that we must invoke non-epistemic values in order to argue that less-direct evidence of quality (e.g. sensitivity and specificity of a detection algorithm for a particular disease) are as valid as unambiguous evidence (e.g. the rate at which patients survive the disease).

Thus, medical image analysis has several issues that exclude it from Betz's (2017) policy-centric perspective of science as medical image analysis research must go through several stages of development and scientific inquiry subject to the aforementioned conditions and separate of policy-makers, forcing the research community to assume both epistemic and non-epistemic values in order to make this research feasible. Given this and Rooney's (1992) arguments that epistemic and non-epistemic are not clearly delineated categories, we want to propose a different way of organising research values in ML for medical image analysis: in terms of their proximity to technical decisions.

### 2.1. How do values effect technical decisions?

For ML in medical image analysis, an analysis of particular research values is complicated as research is not performed by uniform teams with comparable experiences and values, but rather very heterogeneous ones including people with dramatically different backgrounds, from clinicians with a medical background to graduate students with mathematics or computer science backgrounds, which gives them very different notions of value within a singular research project. In the best case, this is harmless; a graduate student can work towards a technical solution guided entirely by considerations such as mathematical elegance or numerical accuracy that hold no interest to their clinical collaborators but do not conflict with them. The same time, the clinical collaborator's values regarding the clinical context do not interfere (but also do not truly guide) the student. However, it could also lead to a more serious misalignment in their research in which the values guiding more technical or lower-level decisions actively conflict with higher-level considerations, such as if said numerical precision requires such excessive computation as to preclude the method from being used in clinic at all. One can easily imagine this leading to conflict, but it also indicates the importance of being explicit and transparent about the value-reasoning taking place, consciously or subconsciously, in a research project.

Sometimes, the values underlying a particular decision are explicitly stated in the associated publication. Birhane et al. (2022) investigated over 100 highly cited ML publications (a small number of which
were associated with medicine) and found 59 or so explicitly stated values. However, values such as “performance” start to stray from the definition of an epistemic value and does not really fit into the non-epistemic variety either. Additionally, Birhane et al. (2022) found that only 4% of papers rigorously connect their methods with the pursuit of a particular value showing that such an analysis is not a prerequisite for publication, even in a field as application-oriented as ML.

In order to bridge the gap between the common notion of values and technical decisions and to side-step questions about whether a value is epistemic or non-epistemic, it would be beneficial to focus on how they relate to each other. According to Ward (2021), there are four senses in which values can relate to decisions: two reason-based (values can motivate decisions and values can justify decisions) and two causal (values can cause decisions, and values can be caused or promoted by decisions). These four avenues are not mutual exclusive, but often implicate each other. For example, a value may justify a particular decision because if such a decision was made, it would causally entail the fulfilment of said value. Alternatively, a value may cause a particular decision because it motivates the decision-maker towards said decision.

Regardless if the relationship is causal or reason-based, it has the potential to be indirect. To illustrate this, imagine two researchers making the same set of technical decisions for a particular clinical problem with the explicit goal of minimising cost. One hopes for said technology to become more available in lower income countries. The other wants to increase the profit margin of their potential company. The values of these two hypothetical researchers could be seen as diametrically opposed but they nevertheless result in the same set of technical decisions via the same intermediate goal. Similarly, researchers may be motivated by the same values but diverge in terms of what technical decisions they make as they may focus on different intermediates. For any connection that is not immediately logically obvious, there must be some intermediate facts or properties to explore. For example, a researcher may want to use a particular method because it is more publishable. Regardless of the method, there must be some underlying context or reasoning regarding what it means for a method to be more or less publishable. Perhaps it is because the method is more widely used. Perhaps the opposite: the method is novel. What is clear from this (bracketing out empirical or theoretical facts) is that the intermediate itself expresses a value: another desirable property that is guiding the technical decisions albeit more directly. Thus, a particular value connects to a particular decision through a logical chain, each link exposing the intermediate reasoning connecting the value and the decision. Some of these chains can also be so common as to go unmentioned, as Birhane et al. (2022) observed:

**The prioritisation of performance values is so entrenched in the field that generic success terms, such as “success”, “progress”, or “improvement” are used as synonyms for performance and accuracy.** However, one might alternatively invoke generic success to mean increasingly safe, consensual, or participatory ML that reckons with impacted communities and the environment. In fact, “performance” itself is a general success term that could have been associated with properties other than accuracy and [being] SOTA [state-of-the-art]. (pg. 179)

Despite the frequency of these patterns, the logical connections are not always immediately obvious and could be incorrect. The latter has lead to the development of frameworks such as metrics (Reinke et al., 2022) or validation (Baxter and Jannin, 2023) flowcharts to help researchers make these technical decisions more consistently based on clinical context. Rather than relying only on conventions, it would be best to explain the reasoning connecting specific values to specific decisions.

The analogy of the value-decision relationship as a physical chain goes a step further — chains can be extended by adding links. That

is, if we can imagine some reason as to why a particular value could reasonably be held, we have found a value that extends our chain. It is also worth noting that, like a physical chain, these logical chains get weaker and less certain with length as they depend increasingly on more and more facts about the world and logical analysis that may not be explicitly or implicitly cognised by a particular agent. (This explains how researchers may make different technical decisions even if they espouse the same values.) Nevertheless, this idea of extending chains gives us a principled manner to explore values by starting with a relatively constrained collection of related technical decisions and extending chains until we find values that adhere more closely to the more commonly-understood notions of values such as those explored previously by Jannin (2021).

It is worth noting that such introspective analysis can be quite cumbersome. However, having a network of said chains “pre-computed” for a particular class of technical decisions can significantly facilitate the process by either providing said reasoning in its entirety or providing a structure on which to add other values and logical chains as required. (Such a network will be presented in Section 4.)

## 2.2. A hierarchy of values for algorithm research

Restricting our view to the field of ML in medical imaging, we propose a hierarchy of values organised by their level of directness with a technical decision. This starts with the most immediate values that directly motivate/cause/imply a decision, then motivations for espousing those values, then motivations for those, and so on until we find terminal values that are frequently held as ends-in-themselves. This hierarchy is not supposed to be discrete but it is useful to have some signpost terms, even if they do not have well-defined boundaries. Ours are:

1. **technical values** immediately relate to the specific technical decisions being made regardless of their concrete context (e.g. speed, memory, elegance, security);

2. **quasi-technical values** involve an immediate side-effect of, or condition for, the implementation/research of the given techniques (e.g. data availability, repeatability, explainability, flexibility);

3. **peri-technical values** relate to the research's context as research rather than just its techniques (e.g. reproducibility, novelty, publishability, commercial potential); and

4. **external values** concern the indirect side-effects of the use of the technology or research in the external world (e.g. clinical benefit, gender/race/geographic fairness, personal status, carbon footprint).

Technical and quasi-technical values are very well known to the ML research community to a point where they made up the vast majority of explicitly stated values observed by Birhane et al. (2022). This is likely due to their more immediate connection to specific technical decisions making it significantly easier to demonstrate how they motivated the specific research with relatively few caveats. Other value analyses such as those of Jannin (2021) and Douglas (2000) focus exclusively on external values which are more well-known to the philosophy of science community and can be discussed in a way that is more technically non-specific. However, one should not consider just a single level of this hierarchy but rather the logical chains spanning across multiple levels. This is because each level is fundamentally incomplete as each either lacks the immediate and unambiguous connection to actual technical decisions or a realistic view of the researcher's cognised values or both.

There appears to be, at least superficially, a relationship between our hierarchy and the difference between epistemic and non-epistemic values. For example, simplicity is both epistemic and technical. Distinctly non-epistemic values such as gender fairness or personal status clearly fall in the external category. However, this pattern does not always hold. For example, reproducibility is frequently taken as a
canonical epistemic value, but lies relatively high on our hierarchy as it has less to do with specific algorithmic decisions and more to do with the process of performing research itself. Outright inverting expectations, values such as security are clearly non-epistemic (having nothing to do with the knowledge generated by the research) but are highly technical due to their immediate relationship with highly technical aspects such as federated learning, resistance to adversarial attacks. etc... Instead, there is a weak conceptual correlation between our hierarchy and the epistemic/non-epistemic divide, rather than strong subsumption.

## 2.3. Ambiguous value terms

Another thing to consider when discussing values is that what seems to be an individual value, upon further inspection, is a placeholder or different name for another value. This can cause significant confusion when two researchers discuss two disjoint concepts using the same word. This is not quite the same thing as a logical chain because in the logical chains discussed above, values encourage or engender each other, rather than being a more or less specific explanation of each other.

Consider the concept of **explainability** or **interpretability** as an example. One could argue that an explanation is a textual description of the process the algorithm follows. Another could argue that an explanation is some indication of what variables or features (or regions of an image) are used and how for a particular decision made by an algorithm given a particular patient. Both are valid definitions of explainability, but focus on different things. The former is *pre hoc* which may make it more useful for things like ensuring an algorithm is reasonable for all patients or finding potential edge cases that would cause it to fail. However, it does not give any information tailored to an individual patient. The latter is *post hoc* in that it allows for quality assurance and traceability for a specific patient but not guidance about how it would work for a future patient.

Another example that has been widely discussed is **fairness** which again is a catch-all term for various disjoint concepts. In their review, Verma and Rubin (2018) discuss some of these including outcome notions (i.e. ML should produce similar predictions for different groups); statistical views (i.e. ML should have similar accuracies for different groups); causal notions (i.e. ML should not use protected characteristics); and inferential notions (i.e. ML should not be able to infer said characteristics). Others, such as (Shelby et al., 2023), focus on fairness from the perspective harm reduction to underprivileged groups when ML is used in practice. These can lead to sometimes conflicting notions of fairness.

To be able to discuss values in this sort of critical manner is important as many of these values (e.g. explainability, fairness, usability, etc...) are explicitly advocated for in multiple ethical AI guidelines (Vandemeulebroucke et al., 2022). Decomposing this ambiguity can be very helpful as different definitions can potentially land on different levels of the hierarchy and thus have differing ways of trickling down into technical decisions. Being able to distinguish between different *instantiations* and different *implications* of value-concepts would also help to clarify when differences are perceptual or fundamental.

## 3. Intermediate representations and the end-to-end vs. separable learning spectrum

Now that we have a definition of values and a way of extending from a set of technical decisions to a series of increasingly high-level values associated with it, we need some inter-related technical decisions to explore. The ones we would like to explore arise from the history of medical image processing and still affect how it is performed today, notably with ML.

Up until recently, addressing complex clinical problems in medical image analysis was universally performed using *pipelines*, series of

computational steps that act on images or imaging-adjacent data. For example, consider post-operative monitoring of a cancer patient. After the patient enters the hospital for an imaging-based follow-up, many things occur: sequences of bio-signals are collected (piezoelectric element voltages for ultrasound, sinograms for CT, k-space for MRI, etc...), and images are reconstructed from these signals. Previously treated tumours are identified and the locations of potential metastases are determined. Then they are segmented and those segmentations measured in terms of diameter/volume/heterogeneity/spiculatedness/etc... Finally, those measurements are used to infer if the cancer is growing or in remission, if the treatment is working or should be adjusted. The conceptual separation of these tasks has been beneficial for the imaging research community, where individual groups can focus on more coherent smaller pieces of the clinical problem.

An *intermediate representation* (IR) is a concept from computer science originally used in compiler design. In this original context, an IR is any data structure that is used to represent the high level code being compiled. The idea behind these IR's that they can sequentially bring the high-level code to the binary machine code used by the computer itself. Construed more broadly, IR's are any data structure that transitions between the input space (e.g. a medical image or signal from a medical imaging device) and the output space (e.g. a diagnosis or clinical decision). In our above example, these IR's could include the segmentation, the measurements taken from said segmentations, etc... Each relates the medical image to the clinical decision in a more structured and less ambiguous way.

Depending on the granularity of perspective, there may be more or fewer of these IR's. At a highly granular perspective, all algorithms include billions that are completely invisible to the user: the bits stored in memory. At a higher level, these are not just bits but rather memory buffers, variables, mathematical trees. Even higher, these memory buffers become images; variables cluster together into patient histories; and mathematical trees resolve themselves into structured reports.

What is notable from pipelines is that humans can directly understand and interact with some of these IR's. An imaging technician looks at the images post-reconstruction to confirm that they are not overburdened by artefacts. A radiologist examines the results of a lesion detection algorithm to confirm that the regions identified as lesions really are so and that no lesion is missed. An oncologist looks at the lesion measurements, ensuring they are coherent with the segmented images and with the diagnosis. These are all *explicit intermediate representations* (EIR's) in the sense that they can be understood by the human clinician and there may also be some burden on the clinician to verify them. This contrasts with implicit ones (such as the aforementioned bits, buffers, and trees) which are not designed to be seen by the user. In the case of ML, some may not even be fully understood by the algorithm designer who made them, resulting in the infamous black-box problem.

There are two distinct philosophies regarding EIR's in ML. *End-to-end learning* minimises them with the goal of basing all IR's on quantitative patterns computationally extracted in the data (Suzuki, 2017). The motivation behind this is that the computer itself may find patterns that a human user would not have (or could not have) considered themselves. The entire computation is thus construed as a complex *input-output function* or *stimulus-response model* leading to its application sometimes being dubbed 'Skinnerian' (Pasquale and Cashwell, 2018) due to its quasi-behaviourist attitude (Knox et al., 2020).

The opposite philosophy, which we call *separable learning*, instead emphasises EIR's. Separable in this context means that the EIR's maintain some conceptual independence, even if they are fundamentally related to each other. For example, even though measuring the volume and surface area of a tumour implies that it has been segmented, we can conceptualise these measurements separately from segmentation which we do when we consider statements such as "in general, tumours with more tortuous boundaries (i.e. they have a higher surface area to volume ratio) are more likely to be malignant than ones that with
<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Example of the end-to-end vs. separable learning spectrum: Subcortical anatomy segmentation for deep brain stimulation.</figcaption></figure>

less tortuous boundaries". Thus, certain relationships can be inferred independently of others (i.e. we can learn to segment tumours separately from learning what properties identify them as malignant or benign) even if the collection of the relationships must be used to apply and validate the framework. For separable learning, human knowledge and interpretation are necessary and heavily constrain the computer's potential solution space to what can be understood by human experts: algorithm developers and clinical users. Related to separable learning is the idea of interpretable models (Rudin, 2019) where a user can understand in a mechanistic way how the model operates. However, for separable learning, the transformations between EIR's might still be opaque but valid. That is, it is likely not necessary for a human user to know exactly how, for example, a medical image reconstruction or segmentation algorithm works in order to confirm that said image or said segmentation is correct for a particular patient.

These two philosophies can be seen as the extremes of a spectrum by varying the number and explicitness of the IR's involved for addressing the same clinical problem. In order to better illustrate IR's and this spectrum, the following subsections are a series of concrete and narrowly construed clinical applications which have been addressed by independent groups in similar ways, distinguishable primarily by the explicitness of their IR's. Each subsection will begin with a brief description of the clinical problem, its inputs, and its outputs, followed by a discussion of three similar papers from which one can interpolate the *end-to-end* vs. separable learning spectrum. In order to show the breadth of this concept, these examples reflect very different medical domains: neurosurgical planning (Section 3.1), cardiac performance measurement (Section 3.2), and breast cancer screening (Section 3.3).

The goal of these examples is not only to show the spectrum nature of these particular design decisions, but also to show the generality of this spectrum in vastly different medical contexts, imaging modalities, and computational tasks. The frameworks also largely use the same ML tools; with the exception of Zhang et al.'s (2021) graph neural network, each method uses convolutional neural networks (CNN's) as the sole learnt components. Thus, the spectrum (unlike supervised vs. unsupervised learning) complements, rather than subsumes, differences in model architectures. This also provides evidence that insights regarding the end-to-end vs. separable learning spectrum can be applied widely to ML research for medical image analysis.

### 3.1. Example: Subcortical brain anatomy segmentation

Deep brain stimulation (DBS) is a surgical treatment for some neurological disorders, notably Parkinson's disease. In DBS, a particular part of the subcortical anatomy is stimulated by an implanted electrode. Thus, an important surgical planning task is to identify this anatomy in pre-operative MRI as to determine an implantation trajectory. The segmentation of these structures is difficult due to their small size (leading to segmentation class imbalance) and similar contrast to other anatomy.

The approaches taken by Dolz et al. (2018), Milletari et al. (2017), and Baxter and Jannin (2022b) are similar in that they rely on CNN's to create a segmented image but differ in terms of the end-to-end vs. separable learning spectrum (Fig. 1).

Starting with the most end-to-end, Dolz et al. (2018) use a patch-based approach in which the image is split into small patches which are segmented by the same CNN then reconstituted into a segmented image. Due to the aforementioned difficulties with small subcortical anatomy segmentation, this approach was applied to larger subcortical areas, but one could easily imagine that with enough training data and appropriate loss function, the problem of segmentation class imbalance could be overcome. Milletari et al. (2017) has the CNN determine both the segmentation and a vector pointing towards the predicted centroid of the object, introducing an EIR. The motivation is that there can be multiple false positives (i.e. isolated, disconnected regions falsely identified as part of the structure) especially for small subcortical anatomy, which are unlikely to have coherent centroid estimates. A post-processing step then filters out these incoherent regions. This side-steps the problem of class imbalance by allowing the network to over-segment, using post-processing to correct for this. Lastly, Baxter and Jannin (2022b) make the centroids even more explicit, training a CNN that iteratively estimates them at increasingly fine resolutions, followed by a second CNN that segments only a small region surrounding the final estimate. As the heavier segmentation network only processes a small part of the image, it is the only framework that allows users to interact with the centroid and receive real-time feedback necessary for said interaction (Baxter et al., 2018). This avoids the problem of segmentation class imbalance entirely by cropping the image being segmented to a size in which the anatomy is no longer so relatively small.

The EIR varied is the centroid of the segmented anatomy. In Dolz et al.'s (2018) framework, this IR is not present whatsoever. In Milletari et al.'s (2017) framework, it is coupled with the segmentation making it more implicit in design. Baxter and Jannin (2022b) render it entirely explicit, estimate it with a separate CNN, and encourage users to interact with it.

### 3.2. Example: Cardiac performance measurement

Diagnosis often relies on measuring a particular anatomical system and its function in some way. For cardiac disorders, the most fundamental measures relate to the functional efficacy of the heart. The benefit of medical imaging is clear in that it allows for anatomically-specific measurements (i.e. ventricular volumes at different moments in the cardiac cycle, the motion profile of different regions of the heart, etc...) which can be more informative and disease-specific than system-wide ones (i.e. pulse, blood pressure, etc...).

Ejection fractions are one of these measurements and many frameworks to compute them rely on the segmentation of the different cardiac chambers. These segmentations form a potential EIR, allowing for users to visualise the chambers being measured and have an impression of the correctness of the metrics. The approaches taken by Lee et al. (2017), Bai et al. (2018), and Abdeltawab et al. (2020) use relatively similar methods, relying again on CNN's applied to cine-MR images, but differ in terms of how they number and purpose, and thus positions on the spectrum (Fig. 2).

Again starting with the most end-to-end, Liu et al. (2020) uses a CNN to directly estimate the left ventricular ejection fraction directly,

<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Example of the end-to-end vs. separable learning spectrum: Cardiac performance measurement.</figcaption></figure>
focusing solely on that singular metric. Thus the volume of the left ventricle is not measured in a traditional sense and other information that could be used as an EIR (e.g. which temporal frame refers to end-systole or end-diastole) is also not estimated, making it very distinctly end-to-end. Liu et al. (2020) advertise this fact in their abstract as a point of novelty for their framework. Bai et al. (2018) took a more traditional approach, using a CNN to segment the various cardiac structures which are then measured separately. This means that their measured properties (notably volume) could be used to directly infer the other aforementioned IR's and finally anatomical metrics. Abdeltawab et al. (2020) simplify the problem by training two networks, one to localise the cardiac region in terms of a moving bounding box and one to segment the structures within, another potential EIR.

These methods differ in terms of the number of EIR's. Liu et al. (2020) use essentially none even for a problem where they seem conceptually necessary. Bai et al. (2018) uses the segmentation of the cardiac chambers and Abdeltawab et al. (2020) adds to that with their approximate location and extent. None explicitly use EIR's for interaction, although one could easily imagine a clinician verifying the quality of the segmented regions or ensuring the right frames are used for the computation.

### 3.3. Example: Breast cancer screening

Mammography-based breast cancer screening involves imaging the breasts using X-rays in order to detect cancerous lesions. Because of the high-throughput nature of screening, automated mammogram evaluation is very desirable. However, the existence of a lesion does not necessarily imply that it is malignant (and warrants treatment) or even that it is cancerous. As biopsy confirmation already requires the attention of an interventionalist as well as different imaging modalities (Soo et al., 2019), the clinical problem for screening is to identify the patient to advance to this step, not to identify the tumour itself. Thus, the input to this particular problem is an X-ray mammogram and the output is a score reflecting the image as a whole.

Li et al. (2020b) use a single CNN to directly classify the entire image according to a single BIRAD score without indicating from where in the image this score arises. Zhang et al. (2021) take a similar approach, classifying the entire image, but include an IR that could help in verifying the framework's conclusion: a CNN-based parcellation of the image into regions acting as vertices in a graph. This graph (and thus the image as a whole) is processed by a graph neural network. The intuitive assumption is that some of these regions represent potential lesions which could potentially be cancerous. Lastly, Ting et al. (2019) makes this intuition more explicit, using one CNN to detect potential lesions and a second to separately classify each one as either benign, malignant, or non-cancerous. The image's overall score is simply whether or not a malignant tumour was detected. These methods again differ primarily in terms of the end-to-end vs. separable learning spectrum (Fig. 3). The specific EIR's of interest here are the identification and segmentation of the lesions themselves, which is intuitive given the pathophysiology of cancer. Although none of the methods explicitly designed this EIR to be interactive (as emphasising

screening throughput generally limits the amount of time a clinician can spend with a single image) it could be used for evaluation purposes or to help communicate results to patients (Soo et al., 2019). The degree of explicitness is important; both Zhang et al. (2020) and Ting et al. (2019) allow for a user to tell if a lesion has been identified as a distinct region but only Ting et al. (2019) show how it contributes to the image's score.

## 4. Research values underlying the end-to-end vs separable learning spectrum

<figure><img src="image_6.png" /><figcaption>Fig. 4. Idealised ML research process.</figcaption></figure>

### 4.1. The idealised ML research process

Our procedure for extending chains of values and our definition of a series of technical decisions related to the end-to-end vs. separable learning spectrum allows for an exploration of what values relate to said decisions. Along with our aforementioned hierarchy, this exploration will also be structured around the idealised ML research process (shown in Fig. 4). This process, although not uniformly followed, consists of multiple steps: define the model (i.e. problem design and architecture specification), find data, train and evaluate the model, possibly introduce the solution into a different research context (i.e. extracting scientific knowledge, translation into clinical practice), publish, then release the algorithms out into the wider world.

The reason why the idealised research process is connected to our proposed hierarchy for this particular spectrum is largely because it too follows a logical structure, although a causal one rather than a cognitive one, which emanates from the same starting point, the solution architecture. As the technical questions characterising our end-to-end vs. separable learning spectrum (those concerning how the ML problem and architecture should be constructed) are included in the first stage of this idealised process, any causal connection in the research process should mirror a logical one separating values from decisions and thus a higher place in our hierarchy. That is, any values directly related to problem definition are, by definition, associated with the first part of the idealised pipeline and are, again by definition, technical research values. Values regarding data collection and annotation are causally influenced by technical decisions (in an idealised world). For example, if we design an architecture that involves image segmentation, at some point, we must collect images annotated with segmentations. This causal connection is mirrored in our hierarchy as if we have a value that encourages certain types of data collection or annotation to occur, that provides us with a reason to prefer particular properties in the techniques applied (i.e. particular technical values) which makes these reasons, by definition, quasi-technical values as they do not encourage a specific technique directly but instead that encourage the technique to have particular side-effects or conditions. An equivalent argument applies for the other stages, noting that the causal/logical separation puts the values associated with later stages in the idealised pipeline occurring at higher levels in the hierarchy, although maybe not enough to justify jumping up a full level. This leaves us finally at the last stage of the research process, real-world use, which is trivially equivalent to external values by the latter's definition.

This correlation is obviously not one-to-one, as there are more parts of the idealised ML research pipeline than there are signpost terms in our hierarchy. Such a correlation also does not hold for all other spectra defining ML, especially those oriented around technical decisions made later in the pipeline. For example, if we created an ML spectrum oriented the leanness of ML research (with ML research focused on

<figure><img src="image_7.png" /><figcaption>Fig. 3. Example of the end-to-end vs. separable learning spectrum: Breast cancer screening.</figcaption></figure>
algorithms that require few resources to research, train, and apply on one side and highly complex research that requires incredibly large databases, significant specialised hardware, and large amount of energy and time on the other) we can easily image some technical values that directly motivate:

* specific low- or high-resource architectures (i.e. the first part of the idealised research pipeline)

* specific low- or high-resource (or human resource) data collection techniques (i.e. the second part of the idealised research process) regardless of the architecture; and

* more or less intensive validation processes which is part of the third part of the idealised research process.

This indicates a de-correlation between the value hierarchy and the idealised ML research process for that particular spectrum as the entry point (i.e. technical values) do not all occur at the beginning of the idealised research process. For more commonly known spectra such as that between supervised and unsupervised learning, the lack of meaningful correlation makes idealised ML research process likely useless for itemising values.

Although the connection between our hierarchy and the idealised ML research process is therefore incidental, but it does provide a more natural way of exploring the domain and encouraging completeness. That is, one can more readily iterate through the values associated with a particular part of the idealised research process than a particular level of the hierarchy of values, being more concrete (at least for ML researchers). A similar process-oriented approach has previously been taken by Ratti and Graves (2022) in order to elucidate the values underlying explainable ML (another spectrum largely characterised by architectural considerations and thus having the same correlation between the idealised research process and the hierarchy of values), showing that it is indeed one method to encourage (although admittedly not guarantee) completeness.

## 4.2. Connecting technical decisions with external values

We now have the conceptual tools necessary for our exploration of the research values related to the end-to-end vs. separable learning spectrum, summarised in Fig. 5. Note that connections in this diagram do not distinguish between the myriad ways that a value could motivate/cause/justify/etc... another value as described by Ward (2021). Such a distinction, however useful, does not add to the goal of this paper. Instead, we use the looser notion of a value engendering another. That is, if A is engendered by B, then someone who espouses B and is aware of particular facts or empirical tendencies about ML, medicine, research, etc... would also espouse A although not necessary vice versa. Most external values in our diagram are shown as a cloud as Jannin (2021) has already explored these values in detail.

Our exploration uncovered a few *value dichotomies* which define trade-offs between values. An example in computer science is the time-memory trade-off which says that one can generally reduce the amount of memory required but at the cost of additional time. These dichotomies largely manifest with one value supporting end-to-end learning and the other supporting separable learning.

In terms of using Fig. 5 in practice, we would encourage it to be used in two ways. One way would be to start at the highest level of research values, then see how they connect to lower level values, keeping track if they are on the left (end-to-end) or right (separable learning) side of the figure. If there is a predominance of values on one side or the other, that indicates that the ML algorithm being developed should adhere more to the corresponding side of the spectrum. The more one-sided the predominance, the more such a decision is supported by the researcher's values. This would be a way of communicating what values ground a particular technical decision. The other way is more introspective. If you find yourself attached to a particular technique or lower-level value, you can trace it to higher levels in the hierarchy using the Figure to see what other values you may subconsciously be assuming. Note

that in both cases, the response you will get will be sensitive to your initial values, showing the importance of wider context to technical decision-making. It may also show particular contradictions where one may be espousing two values on different ends of a dichotomy, showing where a decision is the most sensitive to trade-offs.

### 4.2.1. Problem design and architecture specification

The first stage of addressing a clinical problem is to define it. That is, ideally, we would first define what modalities will be used as input, what the clinically-relevant final output is, and what steps will be performed (or architecture used) in between. This is the spectrum's characteristic manifestation; end-to-end frameworks are essentially defined as those that describe the problem as a function whereas separable ones are those that see it as a series of explicit and human-understandable tasks. The question is then what values motivate either the more abstract, mathematical view or the more concrete task-oriented view.

The principal benefit of detailed decomposition is **problem transparency** in that it requires the human (rather than machine) understanding of the various interacting parts of the solution. Transparency in this sense has a primarily cognitive nature arising from the IR's being more decoupled or isolated and the individual sub-problems understood independently. For example, *parametric models* are equations describing the physical way in which the input image arises from the underlying anatomy (Qayyum et al., 2022), conceptually separating understanding how these parameters manifest in the data (the forward problem) from inferring them (the inverse problem). Generally speaking, sub-problems cannot be fully decoupled (it is through their combined action that the full clinical problem is addressed) but they lead to multiple points in which information can be added to or retrieved from the system. Problem transparency can also be seen as a part of *pre hoc* explainability, giving us insight into how the ML algorithm functions.

One value that clearly favours the abstract, mathematical view (and therefore end-to-end learning) would be **problem simplicity**. By problem simplicity, we mean the brevity with which the specific IR's do not need to be understood by a human (i.e. specific convolution kernel weights, activation functions, number of layers, etc...) while still allowing the problem as a whole to be understood. The paradigmatic case of this would be representing a framework as a singular mathematical functions the details of which are learnt by the computer rather than specified by a human. This makes certain *post hoc* explainability tools feasible such as attention maps<fnref n="1" /> (Hafiz et al., 2021; Huff et al., 2021) which require the entire framework to be a differentiable scalar function. Given the same clinical problem, i.e. the same initial inputs and final outputs, simplicity is the antithesis of transparency; it masks the complexity separating input and output. This is what Grote (2023) refers as the *opacity problem*: as models get larger without EIR's, they become more difficult to understand.

Models may also be usable in multiple clinical problems and the idea of reproducibility within a domain as well as generalisability across domains is one of the key values explicitly held by the majority of ML researchers (Birhane et al., 2022). Separable learning has a superficial advantage here as problems sharing a sub-problem could share models for said sub-problem. This *explicit reuse* (which instantiates the potential offered by problem transparency) is already a part of medical image analysis. For example, brain anatomy segmentation can be seen as both a research target (as in Section 3.1) and an EIR for understanding neurological disorders (Mahlknecht et al., 2010; Cantor-Rivera et al., 2015). Explicit reuse is currently more ubiquitous due to the current predominance of pipelines.

End-to-end learning allows for **implicit reuse**. Several architectures, such as U-Nets (Ronneberger et al., 2015), have been used

<footnote n="1">¹ Attention maps refer to techniques for visualising what input data (e.g. regions of an image) are used the most by an ML model in its prediction.</footnote>
<figure><img src="image_8.png" /><figcaption>Fig. 5. Overall view of the research values differentiating end-to-end and separable learning. The acronyms indicate the section in which the concept is introduced: Problem design (PD - Section 4.2.1), Dataset considerations (DS - Section 4.2.2), Training and evaluation (TE - Section 4.2.3), Knowledge discovery (KD - Section 4.2.4), Clinical translation (CT - Section 4.2.5), Publication (P - Section 4.2.6), and External values (EV - Section 4.2.7).</figcaption></figure>

for various problems (Siddique et al., 2021) sometimes with minimal
changes (Isensee et al., 2018). Model weights can also be reused though
transfer learning (Cheplygina et al., 2019). The advantage of implicit
reuse is that lower requirement for similarity between the original
source problem and the target problem, at the cost of more adaptation,
or fine-tuning. These two types of reuse are not mutually exclusive and
can be applied to almost any model independent of each other.

4.2.2. Dataset considerations

Once the clinical problem is specified, annotated data is required for training or evaluating the ML model or both. The unarguably most important aspect of this is annotation, adding relevant information about the EIR's or desired output. Given the increasing use of Picture Archiving and Communication Systems (PACS) and Electronic Medical Records, more annotated data can be easily collected by clinical collaborators (Alhajeri et al., 2017) allowing for the large increase in data quantity crucial to ML's recent success (Jordan and Mitchell, 2015).

End-to-end learning has a distinct advantage here as its problem simplicity implies that the only annotation needed is the desired final output, such as the patient's diagnosis. Although this still requires a high degree of expertise on the part of the annotator, the annotation is minimal, leading to **annotation efficiency**. Annotation efficiency is a necessary (but not sufficient) condition for high **data availability**, that is, it is one can only collect large amounts of annotated data if sufficient raw data is available and the annotation process is simple and not time-consuming. This may already be a part of the clinical workflow if said annotated data can be added as patients are admitted.

Separable learning requires more annotation per patient, as a ref-
erence is not only needed for the output but also for intermediate
steps. This heightened focus on **annotation informativeness** rather
than efficiency results from problem transparency, as more structured
and fuller annotations are needed to exploit the multiple data extraction
and injection points in a separable model. This can sometimes be
addressed through simulation (Li et al., 2020a) which could help in
training algorithms, but has limited use in their evaluation. Empirically
speaking, the increased information content allows for smaller datasets
to be used, i.e. **data efficiency**. It is not uncommon for medical image
analysis papers to include only hundreds (or even only tens) of patients
rather than thousands. In their review of medical image segmentation
challenges, Antonelli et al. (2022) include ten challenges: only one
has more than 500 images and four have fewer than 100. (One can
imagine segmentation being used for some later downstream task.) This
is in stark contrast to high-profile end-to-end classification tasks, such
as Babenko et al.'s (2022) database of fundus images (145,832 images)
or Lozano's (2017) Medical MNIST dataset (58,954 images).

What may not be obvious is that annotation efficiency and infor-
mativeness are a dichotomy. For example, one can imagine it being
more difficult to accurately diagnose the malignancy of a tumour than
to segment it, although the latter contains more information. However,
they do not represent the same clinical problem. If the problem is
to diagnose cancer, then a diagnosis is still required whether or not
a tumour segmentation is provided. When one considers a clinical
problem with the same inputs and outputs (e.g. a diagnosis) adding
more information about an EIR does not excuse the annotator from specifying the output. The final task cannot be avoided which puts a bound on the potential efficiency, any additions only increasing the annotation time. This is from a per-data-point perspective, as other factors come into play that affect the total annotation effort.

One factor for choosing between annotation efficiency and informativeness is tolerance for uncertain annotations. This uncertainty is most commonly addressed in medical image analysis by collecting multiple independent annotations from one or more users allowing for the quantification of *inter-* and *intra-rater* variability (McGibbon, 2003; Bø et al., 2017), the construction of consensus (Commowick and Warfield, 2010) or “average” annotations (Zhang et al., 2020) for the computer to learn. Involving several medical experts (who may not have dedicated time for tasks outside of clinical workflow) puts severe limits on how much data can be annotated.

Data availability and efficiency both have a connection to external values in terms of privacy, convenience, and accessibility. For example, data availability is associated with “big data” with its related privacy concerns (Tene and Polonetsky, 2011) but also convenience, allowing for some screening procedures (such as ophthalmological screening) to be done more efficiently and remotely with cell phone cameras (Wintergerst et al., 2020). Data efficiency on the other hand is crucial for ML research for smaller patient populations such as those with rare diseases (Schaefer et al., 2020).

### 4.2.3. Training and evaluation considerations

Once the architecture is defined and data collected, the model must be trained to determine its weights and parameters and then evaluated to determine its performance. This is one of the more computationally intensive parts of ML with some algorithms requiring sophisticated hardware or software.

One of the immediate benefits of transparency is that it necessarily allows for different components of separable models to be trained with lower coupling and more mutual independence. This encourages *distributedness* in terms of training and evaluating different parts of the model on separate hardware, potentially at different centres. Distributedness also includes an element of evaluation as each component can be evaluated using metrics catered to that particular sub-problem (Reinke et al., 2022). Techniques such as data augmentation (simulating and even exaggerating potential errors in the pipeline) encourage components to be more mutually robust.

Federated learning has allowed for more distributedness for ML in general, but requires additional computation (exchanging, aggregating, and otherwise reconciling multiple sets of network weights) which adds processing time and limits the independence of the different training sites (Rieke et al., 2020). Federated learning does not imply a preference towards either end of the spectrum as both can readily use it, but it does seem to be necessary for effectively distributing more end-to-end models.

Distributedness extends further than just model training as different parts of a pipeline may be researched by separate groups with diverse expertise. The fact that the human workload can also be distributed plays an important role in collaborative research. This is in opposition to end-to-end learning in which singular models are, for the most part, trained by singular teams. In fact, there does not appear to be a particular value at this stage that clearly motivates end-to-end learning (Section 4.3 discusses some which may seem to be in this category) likely due to the heterogeneity of problems in this domain.

### 4.2.4. Knowledge discovery considerations

Once the model's weights have been determined and the model has been deemed sufficiently performant, there are different avenues that can now be taken. One is to see if some meaningful information can be extracted from it through a process commonly referred to as *knowledge discovery*. That is, if an algorithm consistently learns to model a particular pattern then said pattern likely reflects some objective

fact. However, for this to be extracted from learned IR's, they must have some level of interpretability either inherently or imposed upon it. Separable learning tends towards the former whereas end-to-end models tend towards the latter.

Knowledge discovery in separable learning is facilitated by **logo-centric accessibility**. (We are using Klages' early definition of logo-centrism as the perspective that scientific knowledge is fundamentally textual and the basic entities in said text can or should be considered ontologically real (Klages, 2013). By text, this does not mean a particular language but any human-understandable natural or symbolic language.) For a framework to be logocentrically accessible, a number of its IR's must be textually describable. Given that EIR's are chosen as to be understood by the human user, they must be describable to said users, which implies a certain reliance on text. Knowledge discovery in separable learning thus follows a traditional, well-understood process: properties of the data/EIR's are measured then correlated with each other. For example, several papers have created ML frameworks for neurological knowledge discovery that first register brain images into a common space, segment several predefined structures, measure predefined properties of those structures, and finally correlate them with the patient's disease state (Mahlknecht et al., 2010; Cantor-Rivera et al., 2015). Pixel-based analysis is similar, although it eliminates segmentation in favour of individual locations in the common co-ordinate space (Mechelli et al., 2005; Abe et al., 2010). Another example are the numerous radiomics papers that apply named measures to segmented lesions (Lee et al., 2017). These frameworks fit into textually describable schemas. Connecting this to external values, logocentric accessibility means that discovered knowledge is easier to communicate to the public (allowing for more democratisation) as well as to policymakers who can better make informed healthcare policy than they could with knowledge that is less clear to them.

Setting aside logocentrism-related concerns, it is generally harder to extract knowledge from end-to-end models. Due to their opacity (i.e. the black-box problem), novel information could be interpreted as a potential error or artefact in the model rather than new unexpected knowledge. For example, López-Cabrera et al. (2021) examined several end-to-end style models for detecting COVID19 infection from chest X-rays and found that these models use parts of the image outside of the body: shortcuts rather than new knowledge. This is an obvious case in which the extracted information represents artefacts rather than knowledge, but one can imagine other regions being highlighted (potentially the heart) which would blur this distinction.

Knowledge discovery in end-to-end models is not impossible and can lead to interesting insights. For example, Estudillo-Romero et al. (2022) extracted patterns from CNN's trained on brain MRI using a mix of sensitivity analysis and population neuroimaging. Even then, there are issues of interpretation: a sensitivity towards the intensity of a particular region of the MRI may be taken at face value or it may potentially reflect a sensitivity towards the size of a hyper-intense object located there. This nevertheless shows end-to-end learning's potential for **knowledge flexibility** as the types of extractable patterns are less constrained (Huff et al., 2021) due to the generality and simplicity with which the problem is construed. However, this flexibility also means that the researchers' expectations must be imposed back onto the results such as disregarding a pattern as a shortcut, statistical artefact, or simply irrelevant. In her analysis of the complexity of knowledge extraction from deep learning models, Sullivan (2022) emphasises that the nature of an algorithm as a black-box is not binary but continuous with more “black-boxing” algorithms providing less insight into questions such as “what about patient x leads to the prediction y?” It appears that her concept of “black-boxing” is the inverse of Rudin's (2019) interpretability in which models can be more or less interpretable based on how well a user can understand mechanically how they work and thus give a particular answer. Knowledge flexibility can be directly connected to external values based on the public interest in new and unexpected knowledge pushing the frontiers of science and
generation of new medical hypotheses. An example of this is the idea of "brain age gap" which is the difference between the patient's age and its estimate via neuroimaging (usually with an end-to-end approach). This has furnished many new hypotheses about neurodegenerative disorders (Franke and Gaser, 2019).

### 4.2.5. Clinical translation and workflow considerations

Another optional route is prospective clinical translation in which an algorithm is brought into actual use in a highly controlled environment to determine its practical utility in an ecologically valid context. For medical image analysis, this translation is not a single step, but an incremental process. Those that see prospective use in a single centre must still await larger multi-centre clinical trials. Each of these steps involves regulatory, financial, and ethical barriers that affect values.

Commercial potential is the likelihood of the algorithm making its way onto the market in some capacity either through passing regulatory hurdles for clinical use or as intellectual property to be integrated into other frameworks. In many regions, the regulatory burden is high enough that some element of direct commercialisation is necessary to offset these costs. Because of explicit reuse, separable learning has a generally higher potential as different parts of the model can be protected separately in terms of intellectual property and undergo separate regulatory schemes. The latter is particularly important as more general parts of a separable model might be regulated for different clinical problems, hedging risk through diversification.

Another aspect of clinical translation is how much a framework adheres to pre-existing clinical workflows. One particularly important regulatory aspect is the capability for the framework to adapt to a more conservative approach, called a *fallback mode* (Parvinian et al., 2018). This is often achieved by simply recognising when an error has occurred and then disregarding the model altogether. This fallback mode obviously applies to any system equally well but separable learning allows for less drastic fallback modes via **interactivity**. EIR's can provide opportunities not only for visualising but also correcting errors (Rudin, 2019). For there to be interactivity, there must be an interaction mechanism for some concrete part of the model (Baxter et al., 2018) and thus, higher interactivity is predicated on more transparency, providing opportunities for these mechanisms to exist. Merely exposing IR's to visualisation and editing does what is or is not necessarily make interaction easier and more meaningful as what is or is not considered an IR is a matter of perspective as stated in Section 3. Increasing their number may help when it is already low, but beyond a certain point, one could imagine it becoming burdensome for the user to be responsible for verifying all of them. Nevertheless, this would still be a heightened level of interactivity if the program is designed to make interaction optional rather than mandatory as will be discussed in Section 4.3.2. Thus, IR explicitness, rather number, plays the primary role for interactivity.

### 4.2.6. Publication considerations

Regardless of if the model was used for knowledge discovery, clinical trials, or just as technical research, the results should ideally be published in a scientific journal or technical report. For researchers, there is an expectation to publish to maintain their career. Publishability is not clearly aligned with either side of the spectrum but rather implicates more specific values.

Currently, end-to-end values clearly have a **perception of novelty** and air of disruptiveness as the flexibility of a large neural networks (arising from problem simplicity) lends credence to their capability to address different clinical problems. This novelty gives these models a **wider audience appeal** as more people can more readily appreciate the novelty of a simply defined problem and annotation scheme, even if they do not understand the proposed architecture (Rudin, 2019). By limiting the number of EIR's that must be explained, authors can instead focus on context. This is important for very high-impact journals with restrictive page limits. The words "end-to-end" have also displayed some **keyword capacity**. As shown in Fig. 6, journal papers

<figure><img src="image_9.png" /><figcaption>**Fig. 6.** Prevalence of "end-to-end" as a keyword for ML in medicine papers (PubMed search string: "end-to-end"[Title] AND (neural[Title/Abstract] OR learned[Title/Abstract] OR learning[Title/Abstract] OR deep[Title/Abstract]) NOT (neurorrhapy[Title/Abstract] OR anastomosis[Title/Abstract] OR nerve[Title/Abstract] OR esophagogastrostomy[Title/Abstract]), Search date: January 22, 2025). An equivalent search for "separable learning" yielded only two papers.</figcaption></figure>

increasingly use "end-to-end" in their titles. There is also a growing use of specific architectures' names being used to describe components of new architectures showing how implicit reuse of these architectural components aids their recognisability. Keyword capacity has a mutually supportive relationship with wider audience appeal — the appeal gives a keyword higher recognisability and repeated exposure to a particular keyword increases its familiarity and thus appeal.

Opposite to wider audience appeal is **domain specialisation** due to human aspect of distributedness, allowing researchers to specialise in a particular domain and become more independent of adjacent ones, even when addressing the same clinical problems. The reason for this dichotomy is that scientific journals must either cater towards broader audiences or deeper analysis which is important but less highly accessible. Focusing on domain specialisation has numerous advantages, notably **dedicated jargon**, the particular conceptual tools that can be used with minimal explanation and facilitate communication within a particular community. Dedicated jargon largely requires transparency in terms of the problems and techniques used. This allows for theoretical guarantees about EIR's and a higher level of nuance and standardisation in their evaluation. The importance of dedicated jargon is demonstrated by the existence of highly-focused survey papers on very specific topics such as reconstruction of magnetic resonance angiography with multiple receive coils (Wilson et al., 2004) or ultrasound-MRI registration for brain tumours (Gerard et al., 2021), which do not reflect a singular clinical problem but rather a widely shared sub-problem. Clearly, these indicate the presence of sub-communities with shared knowledge that eases the publication of research within those communities' interests.

### 4.2.7. Real-world use and external values

At this point in the idealised ML research process, the model is now exposed to the wider, uncontrolled world. This could be through various channels, such as the model's code and parameters being released, a company based on it being formed, the knowledge encoded in it being instantiated, or the paper being merely read and cited. It is unsurprising that this is when the external values come into play.

In terms of the researchers themselves, they can be easily motivated by **personal status** and **career advancement** which are ends-in-themselves without any clear ulterior values engendering them. In the case of academia, the two can be largely intertwined as the degree to which one advances in the research community relies on being cited, giving invited talks, chairing events, and other elements that reflect the degree to which personal status is recognised by others. These values are promoted through publications, patents, and the creation of startups: the latter two directly relating to commercial potential. In
terms of publications, both ends of the appeal/specialisation dichotomy
can be supported here. Due to the wide audience appeal of high-
impact journals, they have become implicated in tenure and promotion
decisions, so academics may choose to use methods that are more likely
to be published in higher impact journals solely for the sake of pub-
lishing in said journals. Alternatively academics could take advantage
of specialisation, becoming **the** expert in a field, if narrowly defined.

In terms of the use of the research product itself, **clinical benefit** is a clearly external value which motivates the two values mentioned in the previous section on clinical translation: commercial potential and interactivity. (Of course, clinical benefit as a value is conditioned on the developed models actually reaching clinical use.) Due to these values both favouring separable learning, it is unsurprising that separable frameworks have thus been easier to integrate into the clinic. Aside from clinical integration, commercial potential may also be motivated by **economic activity**, sometimes explicitly requested by governmental funding sources. Both clinical integration and economic activity have clear societal and environmental implications due to the energy, hardware, and human resources as context-specific requirements for further development.

The remaining external values are left represented as a cloud. For
a better understanding of these values, Jannin (2021) analyses them
in the more general context of digital health technology. However,
there are certain values worth explicitly stating due to their more
immediate connection to the quasi-technical values mentioned earlier.
As mentioned earlier, privacy and accessibility are both related to data
availability and data efficiency as the way in which data is collected
can be more or less invasive and sensitive. In terms of knowledge nov-
elty/flexibility, there is a clear link towards the advancement of science
and human knowledge. The degree to which it can be disseminated and
democratised however relies on its logocentric accessibility, its capacity
to be explained through text.

4.3. Other values in medical imaging and ML

In order to make this analysis more complete, we will briefly
examine some other values in ML research in medical imaging that
one may think are associated with a particular side of the end-to-end
vs. separable learning spectrum, but said association is not completely
founded in theory or empirical evidence.

4.3.1. Accuracy, performance, and universality

As noted by Birhane et al. (2022), the primary concern of ML
researchers is undoubtedly accuracy. As many recent models have
become more complex through the addition of more explicit and inter-
pretable components, one might wonder if EIR's may assist in attaining
higher accuracies. Despite this, there always remains the key argument
made by deep learning researchers: it is better (in terms of accuracy)
to allow the computer itself to learn the IR's that are more statistically
optimal than to have a human user specify sub-optimal ones. (For
example, “hand-crafted features” have taken on a pejorative meaning
in the ML community.) However, it is far from certain that a particular
family of ML models is better than another regardless of the nuances
of the clinical problem, patient population, and workflow expectations.
Many of the astounding public claims come from end-to-end models,
but that is due to their perceived novelty and wider audience appeal,
not clinical practice as confirmed by the lack of said models approved
for clinical use (Van der Laak et al., 2021). In terms of universality or
generalisability to different scanners/centres etc...

4.3.2. Automation

Along with interactivity, automation a key aspect of usability. Due
to end-to-end learning's input-output emphasis, it would appear to be
more automated but, as shown in our examples in Section 3, this is
not the case. The potential to interact with EIR's does not imply the
necessity to do so. The key difference is that for end-to-end models

automation is mandatory whereas for separable ones it is a conscious
choice. This choice does however allow for a clinician to use an
algorithm in different ways depending on factors such as correctness
on a particular patient, through interaction mechanisms.

4.3.3. Ease-of-implementation

Although implicit reuse, associated with end-to-end learning, could
be implied by ease-of-implementation (after all, one only needs to reuse
or retrain an architecture for a new problem) it is not always the case
in practice. Even when the same architecture is used, there may be
hyperparameters to set which can be opaque to the algorithm designer,
necessitating some trial-and-error (Bischl et al., 2023). Explicit reuse
does not fully ensure ease-of-implementation either. One must ensure
that inputs to the reused components fall within its operating range and
that the data is sufficiently similar to that used to create the model.

4.3.4. Explainability and interpretability

As stated in Section 2.3, explainability is not a monolithic concept but several, including at minimum notions of pre hoc and post hoc explanations (discussed in Sections 4.2.1 and 4.2.4) which currently favour different ends of our proposed spectrum. There currently something of a bias in favour of separable learning systems for explainability as a whole. However, this might be alleviated in the future with more advanced tools being developed for explainable end-to-end systems, notably those based on counterfactual reasoning. These recent developments still have significant hurdles to overcome, specifically regarding causality (i.e. ensuring the counterfactual is plausible given unknown or unmodelled relationships between input variables) and proximity to the data on which the counterfactual is based (Verma et al., 2020). Such counterfactual techniques have the benefit of being very concrete, often visual, explanations which are currently designed for pre hoc explanation but could be adapted to post hoc ones via structured population studies such as that of Estudillo-Romero et al. (2022).

4.3.5. Fairness

Similar to explainability, fairness is a nuanced collection of different
concepts which can potentially conflict. This nuance is unsurprising as
there are now models that can infer protected characteristics such as
race, gender, and sexual orientation from data where they are invisible
to human doctors (Coyner et al., 2021). Assessing fairness however is
difficult as various metrics of fairness can disagree for specific cases.
Some end-to-end models take a different approach, such as adversarial
de-biasing (Zhang et al., 2018) which penalises models for using pat-
terns that correlate with protected characteristics. Both metric-based
and characteristic-blind approaches also rely on data being annotated
with protected characteristics, which can be problematic as they can
be ambiguous, multifaceted, and co-evolve with public policy resulting
from research decisions about health (Malinowska and Żuradzki, 2022).

4.3.6. Security

Security is a pressing privacy issue given the sensitivity of health
data. In addition to privacy, security also concerns elements of pro-
tection against sabotage and misuse which have other clear social
and societal implications. Generally speaking, security threats arise
from potential unauthorised access to (1) the training data and/or
annotations, (2) the weights of the trained model, or (3) the model itself
during training.

Accessing the training data is the most concerning if done indi-
rectly. For example, Training-data extraction or model inversion attacks
can extract training information from a model's weights using similar
differentiation-based processes as used for training (Zhao et al., 2019).
This would seem to favour separable learning as EIR's often involve
non-differentiable components such as thresholding. This is largely due
to the limitations of current attack techniques, which could evolve in
the future. This could be addressed now by housing the model in a
secure location (similar to how data is currently stored) and controlling
all access. However, this would decrease accessibility, democratisation, and applicability in high throughput or decentralised scenarios that necessitate multiple model instances.

**Adversarial attacks** also require a high degree of access but are designed to deceive rather than to extract data. These attacks use the model's structure and weights to subtly change the input towards a particular erroneous result (Ma et al., 2021). As these attacks are also based on differentiation, EIR's currently thwart them, but this may only be temporary. A crucial difference is that EIR's also expose attacks as they require the attack to maintain both reasonable input and corresponding EIR's. One can still imagine future attacks capable of doing so.

*Data poisoning* implicates the third axis and crucially does not rely on differentiability. It involves injecting patterns into the training data to make the computer learn an association between that particular pattern and the (fake) annotations of the poisoned training data. Later during model application, an attacker can force the model to output a particular result via injecting that same pattern into the input. As with adversarial attacks, these patterns should be invisible (Truong et al., 2020) giving separable learning the same advantage as before, thwarting attacks through visibility, with the same caveat.

These attacks may become a moot point given research on defence mechanisms (Zhao et al., 2019; Truong et al., 2020). These are currently designed for end-to-end models, as they involve augmenting the data in a consistent way during both training and application. These augmentations may require corresponding changes to the EIR's which may be feasible but still complicates separable frameworks. Thus, effective defence mechanisms for end-to-end models might counter separable learning's current advantage.

## 5. On the genealogy of models via related values

As different conceptual tools are developed with difference theoretical guarantees or empirical tendencies, the specifics of how values relate to technical decisions should change. But in addition, values guide which techniques get developed and proliferate. Thus, changes in values and advances in technology are not linear with one implying the other, but should have a more complicated historical relationship.

There has already been some theoretical exploration of this relationship. Dotan and Milli (2019), for example, observed that many shifts in ML research can be explained through research communities' attachment to particular model families (e.g deep learning, probabilistic graphical models, etc...) and to explain this observation, the attachment to particular model types, they invoke an analogy to Kuhnian paradigms and Lakatosian research programmes. (They note a large caveat with this analogy which we also noted in Section 4.2.4: ML models do not carry the same ontological baggage as scientific hypotheses do; they are concerned with solving clinical problems rather than representing something fundamental about medicine. Thus, unlike scientific theories, they are not sensitive to falsification, but rather to other context-dependent concerns.)

Our framework suggests two potential explanations. One is that attachment towards particular model-families is not basic but arises from the attachment to a particular set of technical and quasi-technical values. For example, a strong concern with elegance and theoretical guarantees would necessarily lead a particular researcher towards the attachment to simpler models in which theoretical explorations are readily made. Although this perspective does explain the historical attachment to particular models, it still leaves unexplained the attachment to those values. An alternative (and more fruitful explanation) comes from constantly held values related to academic publishing. Attachment to a domain in which one is already deeply familiar with accepted techniques and theory would allow for faster and easier publication. However, due to the selective nature of journals, some perceived novelty is also necessary, forcing one outside of this well-explored

domain. One must also compare the model against the “state-of-the-art” which requires researchers to not only be aware of developments in the field but actively capable of reproducing them. This again leads to the same observations as Dotan and Milli (2019) but explained through more mundane publishability concerns. The constancy of publishability forces the balance between its intermediate values to change, moving down the chain towards technical decisions. As these techniques enter the wider world, they will have an effect on external values (as per Ward's (2021) description of some values being a causal result of technical decisions) albeit at a much slower rate due to the temporal delay of ML research leaving academia.

The history of the end-to-end vs. separable learning spectrum itself, despite its clear relationship to the pipeline paradigm discussed in Section 3, is much more clear even if it has only recently been articulated. As noted by Dotan and Milli (2019), earlier ML approaches were limited in terms of computational power and data availability, both of which encouraged smaller, simpler models to be used. Even certain problems that we now uniformly associate with end-to-end models, such as medical image synthesis where images in particular modalities are created from scratch rather than collected from a scanner (Yu et al., 2020), have pipeline-based precursors (Chen et al., 2002). If one assumes that the clinical problems of the recent past are equally complex as those of today, this increase in data and computation power would be the key for moving away from separable approaches with simpler, chained models, towards more complex but singular end-to-end approaches, and this movement allows for a degree of variability to exist.

## 6. Conclusions

There is ample literature illustrating the concrete benefits of discussing values during scientific decision making regarding medicine (Edwards et al., 1998; Douglas, 2000; Elliott and McKaughan, 2009; Lalumera and Fanti, 2019; Malinowska and Żuradzki, 2022) and science more generally (Popper, 1963; Rooney, 1992; Betz, 2013, 2017), but at this stage, it is worth taking account of the frequency with which reasoning about values takes place in the literature regarding ML in medical image analysis. We have already pointed out that discussions of value-reasoning do account in technical research papers (Birhane et al., 2022) although this is admittedly in to a smaller degree than research papers explicitly about values in ML in medicine (Jannin, 2021; Souza et al., 2023; Topaloglu et al., 2021). There has been a proliferation in recent years of frameworks for ethical artificial intelligence in medicine and medical imaging. Vandemeulebroucke et al. (2022) in their literature review compared a startling 57 ethical AI frameworks, indicating that ML researchers are indeed discussing values even if they do not frequently make it into technical papers. Thus, one can assume that such reasoning is beneficial given the sheer interest in it.

The missing aspect, as stated in Section 2, is the connection between specific values and specific technical decisions. This is not only an issue for medicine, but more broadly as Prem (2023), in his review of over 100 ethical AI frameworks, observed that “many of them are too abstract for being easily translated into concrete designs for AI systems”. The goal of our paper is to help lay the theoretical foundations for exactly this kind of translation. This article makes three contributions:

1. the elucidation of the indirect nature in which values affect technical decisions, organising them by directness;

2. the identification of the end-to-end vs. separable learning spectrum as a collection of decisions regarding the use of explicit intermediate representations in structuring ML algorithms for medical image analysis; and

3. an exploration of the values associated with either end of this spectrum beginning with the characterisation of these technical decisions, then advancing through layers of increasing indirect and high-level values.
This culminates in the construction of Fig. 5 which shows the logical
chains connecting a plethora of different values at different levels of the
hierarchy to each side of end-to-end vs. separable learning spectrum.

Our analysis reveals many of the same technical values identified empirically by Birhane et al. (2022) and dovetails with Jannin's (2021) analysis, illustrating that ML researchers are not ambivalent towards explicitly incorporating research values into their own research. This may indicate that our analysis might be generalisable to other fields that make use of ML. One constraint is the reliance on the existence and variability of EIR's. Certain fields display this. Natural language processing currently has models ranging from those based on representing words, grammatical structures, intentions, etc... to ones that do not even segment individual words, let alone grammatical structures. Other fields lack this variability. For example, ML for self-driving cars currently makes heavy use of explicit intermediate representations: they search for vehicles, signage, pedestrians etc... as well as represent traffic laws and best practices, before determining the physical action the car should take. ML in finance is the opposite — EIR's are exceedingly rare.

Similar analyses may be possible for other groups of technical decisions within medical image analysis. Future articles could examine the values behind the conceptual tools currently driving ML research, such as open databases and federated learning, and some have already been published (Souza et al., 2023; Topaloglu et al., 2021). We hope that this article can introduce those in the medical image processing community to how to understand research values in a structured manner as well as to introduce members of the philosophical community to some of the nuances specific to ML in medical image analysis. The intersection of these communities will hopefully yield deeper understandings into how ML is applied in medicine.

CRediT authorship contribution statement

**John S.H. Baxter:** Writing – review & editing, Writing – original draft, Investigation, Conceptualization. **Roy Eagleson:** Writing – original draft, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

No data was used for the research described in the article.

References

Abdeltawab, H., Khalifa, F., Taher, F., Alghamdi, N.S., Ghazal, M., Beache, G., Mohamed, T., Keynton, R., El-Baz, A., 2020. A deep learning-based approach for automatic segmentation and quantification of the left ventricle from cardiac cine MR images. Comput. Med. Imaging Graph. 81, 101717.

Abe, O., Takao, H., Gonoi, W., Sasaki, H., Murakami, M., Kabasawa, H., Kawaguchi, H., Goto, M., Yamada, H., Yamasue, H., et al., 2010. Voxel-based analysis of the diffusion tensor. Neuroradiology 52 (8), 699-710.

Alhajeri, M., Aldosari, H., Aldosari, B., 2017. Evaluating latest developments in PACS and their impact on radiology practices: a systematic literature review. Informatics Med. Unlocked 9, 181-190.

Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B., Ronneberger, O., Summers, R.M., et al., 2022. The medical segmentation decathlon. Nat. Commun. 13 (1), 1-13.

Babenko, B., Mitani, A., Traynis, I., Kitade, N., Singh, P., Maa, A.Y., Cuadros, J., Corrado, G.S., Peng, L., Webster, D.R., et al., 2022. Detection of signs of disease in external photographs of the eyes via deep learning. Nat. Biomed. Eng. 1-14.

Bai, W., Sinclair, M., Tarroni, G., Oktay, O., Rajchl, M., Vaillant, G., Lee, A.M., Aung, N., Lukaschuk, E., Sanghvi, M.M., et al., 2018. Automated cardiovascular magnetic resonance image analysis with fully convolutional networks. J. Cardiovasc. Magn. Reson. 20 (1), 1-12.

Baxter, J.S.H., 2022. Separable vs. End-to-end learning: A critical examination of learning paradigms. In: Workshop on the Ethical and Philosophical Issues in Medical Imaging. Springer, pp. 26-37.

Baxter, J.S.H., Gibson, E., Eagleson, R., Peters, T.M., 2018. The semiotics of medical image segmentation. Med. Image Anal. 44, 54-71.

Baxter, J.S.H., Jannin, P., 2022a. Bias in machine learning for computer-assisted surgery and medical image processing. Comput. Assist. Surg. 27 (1), 1-3.

Baxter, J.S.H., Jannin, P., 2022b. Combining simple interactivity and machine learning: a separable deep learning approach to subthalamic nucleus localization and segmentation in MRI for deep brain stimulation surgical planning. J. Med. Imaging 9 (4), 045001.

Baxter, J.S.H., Jannin, P., 2023. Validation in the age of machine learning: A framework for describing validation with examples in transcranial magnetic stimulation and deep brain stimulation. Intelligence- Based Med. 100090.

Betz, G., 2013. In defence of the value free ideal. Eur. J. Philos. Sci. 3, 207-220.

Betz, G., 2017. Why the argument from inductive risk doesn't justify incorporating non-epistemic values in scientific reasoning. In: Current Controversies in Values and Science. Routledge New York, NY, London, pp. 94-110.

Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., Bao, M., 2022. The values encoded in machine learning research. In: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. pp. 173-184.

Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A.-L., et al., 2023. Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. Wiley Interdiscip. Reviews: Data Min. Knowl. Discov. 13 (2), e1484.

Bluhm, R., 2017. Inductive risk and the role of values in clinical trials. Explor. Inductive Risk: Case Stud. Values Sci. 193-212.

Bø, H.K., Solheim, O., Jakola, A.S., Kvistad, K.A., Reinertsen, I., Berntsen, E.M., 2017. Intra-rater variability in low-grade glioma segmentation. J. Neurooncol. 131 (2), 393-402.

Cantor-Rivera, D., Khan, A.R., Goubran, M., Mirsattari, S.M., Peters, T.M., 2015. Detection of temporal lobe epilepsy using support vector machines in multi-parametric quantitative MR imaging. Comput. Med. Imaging Graph. 41, 14-28.

Caton, S., Haas, C., 2024. Fairness in machine learning: A survey. ACM Comput. Surv. 56 (7), 1-38.

Chen, J.Z., Pizer, S.M., Chaney, E.L., Joshi, S., 2002. Medical image synthesis via Monte Carlo simulation. In: Medical Image Computing and Computer-Assisted Intervention—MICCAI 2002: 5th International Conference Tokyo, Japan, September 25-28, 2002 Proceedings, Part I 5. Springer, pp. 347-354.

Cheplygina, V., de Bruijne, M., Pluim, J.P., 2019. Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Med. Image Anal. 54, 280-296.

Comrowick, O., Warfield, S.K., 2010. Incorporating priors on expert performance parameters for segmentation validation and label fusion: a maximum a posteriori STAPLE. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 25-32.

Coyner, A.S., Singh, P., Brown, J.M., Ostmo, S., Chan, R., Chiang, M.F., Kalpathy-Cramer, J., Campbell, J.P., 2021. Not color blind: AI predicts racial identity from black and white retinal vessel segmentations. arXiv preprint arXiv:2109.13845.

Dolz, J., Desrosiers, C., Ayed, I.B., 2018. 3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study. NeuroImage 170, 456-470.

Dotan, R., Milli, S., 2019. Value-laden disciplinary shifts in machine learning. arXiv preprint arXiv:1912.01172.

Douglas, H., 2000. Inductive risk and values in science. Philos. Sci. 67 (4), 559-579.

Eagleson, R., 2024. Implicit Bias in Human and Artificial Intelligence. Inspiring Diversity in Science Technology Engineering and Mathematics, Western University, Workshop Presentation URL https://www.inspiringdiversityinstem.com/2024-workshops.

Edwards, S.J., Lilford, R., Hewison, J., 1998. The ethics of randomised controlled trials from the perspectives of patients, the public, and healthcare professionals. BMj 317 (7167), 1209-1212.

Elliott, K.C., 2017. A Tapestry of Values: An Introduction to Values in Science. Oxford University Press.

Elliott, K.C., McKaughan, D.J., 2009. How values in scientific discovery and pursuit alter theory appraisal. Philos. Sci. 76 (5), 598-611.

Estudillo-Romero, A., Haegelen, C., Jannin, P., Baxter, J.S.H., 2022. Voxel-based diktometry: Combining convolutional neural networks with voxel-based analysis and its application in diffusion tensor imaging for Parkinson's disease. Hum. Brain Mapp. 43 (16), 4835-4851.

Franke, K., Gaser, C., 2019. Ten years of BrainAGE as a neuroimaging biomarker of brain aging: what insights have we gained? Front. Neurol. 789.

Gefenas, E., Cekanauskaita, A., Tuzaite, E., Dranseika, V., Characiejus, D., 2011. Does the “new philosophy” in predictive, preventive and personalised medicine require new ethics? EPMA J. 2 (2), 141-147.

Gerard, I.J., Kersten-Oertel, M., Hall, J.A., Sirhan, D., Collins, D.L., 2021. Brain shift in neuronavigation of brain tumors: an updated review of intra-operative ultrasound applications. Front. Oncol. 10, 618837.

Grote, T., 2021. Randomised controlled trials in medical AI: ethical considerations. J. Med. Ethics.
Grote, T., 2023. The allure of simplicity: On interpretable machine learning models in healthcare. Philos. Med. 4 (1).

Hafiz, A.M., Parah, S.A., Bhat, R.U.A., 2021. Attention mechanisms and deep learning for machine vision: A survey of the state of the art. arXiv preprint arXiv:2106.07550.

Huff, D.T., Weisman, A.J., Jeraj, R., 2021. Interpretation and visualization techniques for deep learning models in medical imaging. Phys. Med. Biol. 66 (4), 04TR01.

Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., Maier-Hein, K.H., 2018. No new-net. In: International MICCAI Brainlesion Workshop. Springer, pp. 234–244.

Jannin, P., 2021. Towards responsible research in digital technology for health care. arXiv preprint arXiv:2110.09255.

Jordan, M.I., Mitchell, T.M., 2015. Machine learning: Trends, perspectives, and prospects. Science 349 (6245), 255–260.

Klages, L., 2013. The Biocentric Worldview. Arktos.

Knox, J., Williamson, B., Bayne, S., 2020. Machine behaviourism: Future visions of 'learnification' and 'dataification' across humans and digital technologies. Learn. Media Technol. 45 (1), 31–45.

Lalumera, E., Fanti, S., 2019. Randomized controlled trials for diagnostic imaging: Conceptual and practical problems. Topoi 38 (2), 395–400.

Lee, G., Lee, H.Y., Park, H., Schiebler, M.L., van Beek, E.J., Ohno, Y., Seo, J.B., Leung, A., 2017. Radiomics and its emerging role in lung cancer research, imaging biomarkers and clinical management: state of the art. Eur. J. Radiol. 86, 297–307.

Li, D., Kar, A., Ravikumar, N., Frangi, A.F., Fidler, S., 2020a. Federated simulation for medical imaging. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 159–168.

Li, C., Xu, J., Liu, Q., Zhou, Y., Mou, L., Pu, Z., Xia, Y., Zheng, H., Wang, S., 2020b. Multi-view mammographic density classification by dilated and attention-guided residual learning. IEEE/ ACM Trans. Comput. Biology Bioinform. 18 (3), 1003–1013.

Liu, Z., Zhang, Y., Li, W., Li, S., Zou, Z., Chen, B., 2020. Multislice left ventricular ejection fraction prediction from cardiac MRIs without segmentation using shared SpTDenNet. Comput. Med. Imaging Graph. 86, 101795.

López-Cabrera, J.D., Orozco-Morales, R., Portal-Díaz, J.A., Lovelle-Enríquez, O., Pérez-Díaz, M., 2021. Current limitations to identify COVID-19 using artificial intelligence with chest X-ray imaging. Heal. Technol. 11 (2), 411–424.

Lozano, A.P., 2017. Medical MNIST classification. GitHub Repository https://github.com/apolanco3225/Medical-MNIST-Classification.

Ma, X., Niu, Y., Gu, L., Wang, Y., Zhao, Y., Bailey, J., Lu, F., 2021. Understanding adversarial attacks on deep learning based medical image analysis systems. Pattern Recognit. 110, 107332.

Mahlknecht, P., Hotter, A., Hussl, A., Esterhammer, R., Schocke, M., Seppi, K., 2010. Significance of MRI in diagnosis and differential diagnosis of Parkinson's disease. Neurodegener. Dis. 7 (5), 300–318.

Malinowska, J.K., Zuradzki, T., 2022. Towards the multileveled and processual conceptualisation of racialised individuals in biomedical research. Synthese 201 (1), 11.

McGibbon, C.A., 2003. Inter-rater and intra-rater reliability of subchondral bone and cartilage thickness measurement from MRI. Magn. Reson. Imaging 21 (7), 707–714.

Mechelli, A., Price, C.J., Friston, K.J., Ashburner, J., 2005. Voxel-based morphometry of the human brain: methods and applications. Curr. Med. Imaging 1 (2), 105–113.

Mienye, I.D., Swart, T.G., Obaido, G., 2024. Fairness metrics in AI healthcare applications: a review. In: 2024 IEEE International Conference on Information Reuse and Integration for Data Science. IRI, IEEE, pp. 284–289.

Milletari, F., Ahmadi, S.-A., Kroll, C., Plate, A., Rozanski, V., Maiostre, J., Levin, J., Dietrich, O., Ertl-Wagner, B., Bötzel, K., et al., 2017. Hough-CNN: deep learning for segmentation of deep brain regions in MRI and ultrasound. Comput. Vis. Image Underst. 164, 92–102.

Parvinian, B., Scully, C., Wiyor, H., Kumar, A., Weininger, S., 2018. Regulatory considerations for physiological closed-loop controlled medical devices used for automated critical care: food and drug administration workshop discussion topics. Anesth. Analg. 126 (6), 1916.

Pasquale, F., Cashwell, G., 2018. Prediction, persuasion, and the jurisprudence of behaviourism. Univ. Tor. Law J. 68 (supplement 1), 63–81.

Popper, K.R., 1963. Science as falsification. Conjectures Refutations 1 (1963), 33–39.

Prem, E., 2023. From ethical AI frameworks to tools: a review of approaches. AI Ethics 3 (3), 699–716.

Qayyum, A., Ilahi, I., Shamshad, F., Boussaid, F., Bennamoun, M., Qadir, J., 2022. Untrained neural network priors for inverse imaging problems: A survey. IEEE Trans. Pattern Anal. Mach. Intell.

Ratti, E., Graves, M., 2022. Explainable machine learning practices: opening another black box for reliable medical AI. AI Ethics 2 (4), 801–814.

Reinke, A., Maier-Hein, L., Christodoulou, E., Glocker, B., Scholz, P., Isensee, F., Kleesiek, J., Kozubek, M., Reyes, M., Riegler, M.A., et al., 2022. Metrics reloaded: a new recommendation framework for biomedical image analysis validation. In: Medical Imaging with Deep Learning.

Rieke, N., Hancox, J., Li, W., Milletari, F., Roth, H.R., Albarqouni, S., Bakas, S., Galtier, M.N., Landman, B.A., Maier-Hein, K., et al., 2020. The future of digital health with federated learning. NPJ Digit. Med. 3 (1), 1–7.

Ronneberger, O., Fischer, P., Brox, T., 2015. U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 234–241.

Rooney, P., 1992. On values in science: Is the epistemic/non-epistemic distinction useful? In: PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association, Vol. 1992, No. 1. Cambridge University Press, pp. 13–22.

Rudin, C., 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1 (5), 206–215.

Savard, J., 2013. Personalised medicine: a critique on the future of health care. J. Bioethical Inq. 10 (2), 197–203.

Schaefer, J., Lehne, M., Schepers, J., Prasser, F., Thun, S., 2020. The use of machine learning in rare diseases: a scoping review. Orphanet J. Rare Dis. 15 (1), 1–10.

Shelby, R., Risman, S., Henne, K., Moon, A., Rostamzadeh, N., Nicholas, P., Yilla-Akbari, N., Gallegos, J., Smart, A., Garcia, E., et al., 2023. Sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction. In: Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. pp. 723–741.

Siddique, N., Paheding, S., Elkin, C.P., Devabhaktuni, V., 2021. U-net and its variants for medical image segmentation: A review of theory and applications. Ieee Access 9, 82031–82057.

Soo, M.S., Shelby, R.A., Johnson, K.S., 2019. Optimizing the patient experience during breast biopsy. J. Breast Imaging 1 (2), 131–138.

Souza, R., Stanley, E.A., Forkert, N.D., 2023. On the relationship between open science in artificial intelligence for medical imaging and global health equity. In: Workshop on Clinical Image-Based Procedures. Springer, pp. 289–300.

Sullivan, E., 2022. Understanding from machine learning models. British J. Philos. Sci.

Suzuki, K., 2017. Overview of deep learning in medical imaging. Radiol. Phys. Technol. 10 (3), 257–273.

Tene, O., Polonetsky, J., 2011. Privacy in the age of big data: a time for big decisions. Stan. L. Rev. Online 64, 63.

Ting, F.F., Tan, Y.J., Sim, K.S., 2019. Convolutional neural network improvement for breast cancer classification. Expert Syst. Appl. 120, 103–115.

Topaloglu, M.Y., Morrell, E.M., Rajendran, S., Topaloglu, U., 2021. In the pursuit of privacy: the promises and predicaments of federated learning in healthcare. Front. Artif. Intell. 4, 746497.

Truong, L., Jones, C., Hutchinson, B., August, A., Praggastis, B., Jasper, R., Nichols, N., Tuor, A., 2020. Systematic evaluation of backdoor data poisoning attacks on image classifiers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 788–789.

Van der Laak, J., Litjens, G., Ciompi, F., 2021. Deep learning in histopathology: the path to the clinic. Nature Med. 27 (5), 775–784.

Vandemeulebroucke, T., Denier, Y., Mertens, E., Gastmans, C., 2022. Which framework to use? A systematic review of ethical frameworks for the screening or evaluation of health technology innovations. Sci. Eng. Ethics 28 (3), 26.

Verma, S., Dickerson, J., Hines, K., 2020. Counterfactual explanations for machine learning: A review, vol. 2. p. 1, arXiv preprint arXiv:2010.10596.

Verma, S., Rubin, J., 2018. Fairness definitions explained. In: Proceedings of the International Workshop on Software Fairness. pp. 1–7.

Ward, Z.B., 2021. On value-laden science. Stud. Hist. Philos. Sci. Part A 85, 54–62.

Wilson, G.J., Hoogeveen, R.M., Willinek, W.A., Muthupillai, R., Maki, J.H., 2004. Parallel imaging in MR angiography. Top. Magn. Reson. Imaging 15 (3), 169–185.

Wintergerst, M.W., Jansen, L.G., Holz, F.G., Finger, R.P., 2020. Smartphone-based fundus imaging—where are we now? Asia- Pac. J. Ophthalmol. 9 (4), 308–314.

Yu, B., Wang, Y., Wang, L., Shen, D., Zhou, L., 2020. Medical image synthesis via deep learning. Deep. Learn. Med. Image Analysis: Challenges Appl. 23–44.

Zhang, M., Gao, J., Lyu, Z., Zhao, W., Wang, Q., Ding, W., Wang, S., Li, Z., Cui, S., 2020. Characterizing label errors: confident learning for noisy-labeled image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 721–730.

Zhang, B.H., Lemoine, B., Mitchell, M., 2018. Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. pp. 335–340.

Zhang, Y.-D., Satapathy, S.C., Guttery, D.S., Görriz, J.M., Wang, S.-H., 2021. Improved breast cancer classification through combining graph convolutional network and convolutional neural network. Inf. Process. Manage. 58 (2), 102439.

Zhao, J., Chen, Y., Zhang, W., 2019. Differential privacy preservation in deep learning: Challenges, opportunities and solutions. IEEE Access 7, 48901–48911.