Data Models for Dataset Drift Controls in Machine Learning With Optical Images
Authors: Luis Oala
Marco Aversa
Gabriel Nobis
Kurt Willis
Yoan Neuenschwander
Michèle Buck
Christian Matek
Jerome Extermann
Enrico Pomarico
Wojciech Samek
Roderick Murray-Smith
Christoph Clausen
Bruno Sanguinetti
Venue: Accepted by TMLR ()

Abstract: Camera images are ubiquitous in machine learning research. They also play a central role in the delivery of important public services spanning medicine or environmental surveying. However, the application of machine learning models in these domains has been limited because of robustness concerns. A primary failure mode are performance drops due to differences between the training and deployment data. While there are methods to prospectively validate the robustness of machine learning models to such dataset drifts, existing approaches do not account for explicit models of machine learning's primary object of interest: the data. This limits our ability to study and understand the relationship between data generation and downstream machine learning model performance in a physically accurate manner. In this study, we demonstrate how to overcome this limitation by pairing traditional machine learning with physical optics to obtain explicit and differentiable data models. We demonstrate how such data models can be constructed for image data and used to control downstream machine learning model performance related to dataset drift. The findings are distilled into three applications. First, drift synthesis enables the controlled generation of physically faithful drift test cases to power model selection and targeted generalization. Second, the gradient connection between machine learning task model and data model allows advanced, precise tolerancing of task model sensitivity to changes in the data generation. These drift forensics can be used to precisely specify the acceptable data environments in which a task model may be run. Third, drift optimization opens up the possibility to create drifts that can help the task model learn better faster, effectively optimizing the data generating process itself to support the downstream machine vision task. This is an interesting upgrade to existing imaging pipelines which traditionally have been optimized to be consumed by human users but not machine learning models. Alongside the data model code we release two datasets to the public that we collected as part of this work. In total, the two datasets, Raw-Microscopy and Raw-Drone, comprise 1,488 scientifically calibrated reference raw sensor measurements, 8,928 raw intensity variations as well as 17,856 images processed through twelve data models with different configurations. A guide to access the open code and datasets is available at https://github.com/aiaudit-org/raw2logit.


Published in Transactions on Machine Learning Research (05/2023)
Data Models for Dataset Drift Controls in Machine Learning With Optical Images
Luis Oalaj
muis.oolo@ddopphooooccm
Fraunhofer HHT and Dotphoton AG
Marco Aversa*
marco.eerraakddhhhotooocoom
Dotphoton AG and University of Glasgow
Gabriel Nobis
Fraunhofer HHT
Kurt Willis Frawnhofer HI!
kurt, burt.uullisshhiffraabbofee.dddd
Yoan Neuenschwander HEPIA/HES-SO
Michèle Buck
michhlee.kgnclSttam..dd
Klinskum rechls der Isar
Christian Matek
Helmholtz Zentrum Munich
Jérôme Extermann HEPIA/HES-SO
jeronee.cttermaannnhhheee.ee
Enrico Pomarico HEPIA/HES-SO
enrio..omarrio@@eese...
Wojeicch Samek Fraunhofer I!II
wwjjceehhsammckkhhttffaaaau.
Roderick Murray-Smith University of Glasgow
roderick. seeervdd-wurrry---iiiillggggggg
Christoph Clausen Dolphoton AG
Bruno Sanguinetti Dolphoton AG
brw...annyiieeti@@lohhhcccocmmm
Reviewed on OpenReview: https: openreview net/ forum? id= 14 I4IKGG2FJ2
"Equal contribution

---

Published in Transactions on Machine Learning Research (05/2023)
Abstract
Camera images ubiquitous in machine learning research. They also play central role in the delivery of important public services spanning medicine or environmental surveying. However, the application of machine learning models in these domains has been limited because of robustness concerns. A primary failure mode are performance drops due to differences between the training and deployment data. While there are methods to prospec- tively validate the robustness of machine learning models to such dataset drifts, existing approaches do not account for explieit models of machine learning's primary object of interest: the data. This limits our ability to study and understand the relationship between data generation and downstream machine learning model performance in physically accurate manner. In this study, we demonstrate how to overcome this limitation by pairing traditional machine learning with physical optics to obtain explicit and differentiable data models. We demonstrate how such data models can be constructed for image data and used to control downstream machine learning model performance related to dataset drift. The findings are distilled into three applications. First, drift synthesis enables the controlled generation of physically faithful drift test cases to power model selection and targeted generalization. Second, the gradient connection between machine learning lask model and data model allows advanced, precise tolerancing of task model sensitivity to changes in the data generation. These drift forensics can be used to precisely specify the acceptable data environments in which a task model may be run. Third, drift optimization opens up the possibility to create drifts that can help the task model learn better faster, effectively optimizing the data generating process itself to support the downstream machine vision task. This is an interesting upgrade to cxisting imaging pipelines which traditionally have been optimized to be consumed by human users but not machine learning models. The data models require access to raw sensor images as commonly processed at scale in industry domains such as microscopy, biomedicine, autonomous vehicles or remote semsing. Alongside the data model code we release two datasets to the public that we collected as part of this work. In total, the two datasets, Raw-Microscopy and Raw-Drone, comprise 1,488 scientifically calibrated reference raw sensor measurements, 8,928 raw intensity variations as well as 17,856 images processed through (welve daia models with different configurations. A guide to access the open code and datasets is available at httph: bectp://githhb. com/ coom//iauditoor/raa/lla
1 Introduction
In this study we demonstrate how explicit data models for images can be constructed and then be used for advanced dataset drift. controls machine learning workflows. We connect raw image data, differentiable data models and the standard machine learning pipeline. This combination enables novel, physically faithful validation protocols that can be used towards intended use specifications of machine learning systems, necessary pre-requisite for the use of any technology in many application domains such as medicine autonomous vehicles.
Camera image data are a staple of machine learning research, from the early proliferation of neural networks on MNIST 1 to leaps in deep learning on CIFAR and ImageNet b or high-dimensional generative models g Camera images also play an important role the delivery of various high-impact public and commercial services. Unsurprisingly, the exceptional capacity of deep supervised learning has inspired great. imagination to automate enhance such services. During the 2010s, 'deep learning for ...' rang loud in most application domains under the sun, and beyond 10, spanning medicine and biology (microscopy for
cell detection [ [4, histopathology 15 16 opthalmology 17 19, malaria detection 20 28 geospatial
modelling (climate 24 26 precision farming 27 29
pollution detection
30 32
and more.
However, the excitement was reined in by calls for caution. Machine learning systems exhibit particular failure modes that are contingent on the makeup of their inputs 33
85 Many findings from the machine learning robustness literature confirm supervised learning's tremendous capacity for identifying features in the training
2

---

Published in Transactions on Machine Learning Research (05/2023)
inputs that are correlated with the true labels 36:00 But these findings also point to flipside of this capacity: the sensitivity of the resulting machine learning model's performance to changes both large and small - in the input data, Because this dependency touches on generalization, a summum bonum of machine learning, the implications have been studied across most of its many sub-disciplines including robustness validation AT 14 formal model verification 55 uncertainly quantification
out-of-distribution semi- S 00 and self-supervised learning 991 92, , learning theory and optimization
72 82
detection
93
190.
federated learning
99
or compression 100
102]
among others.
We refer to the mechanism underlying changes in the input data as dataset drif Formally, characterize it as follows. Let (Xnaw. Y)w. :2 -2 R.W. x x be the raw sensor data generating random variable on some probability space FP), for example with J = 1, for a classification task. Raw inputs &RAW are in a data state before further processing is applied, in our case photons captured by the pixels of a camera sensor as displayed in the outputs of the "Measurement" block in Figure [ processed by a data model Prroc:R itself or other downst data processing pipelines, to produce a processed view v 4 Proce (#RAA) of the data as illustrated in the output of the "Data model" block in Figure ! This processed view v could for example be the finished RGB image, the image data state that most machine lcarning rescarchers typically work with to train a task model Trask : R.C..W + y Thus, in the conventional machine learning setting we obtain V = Prrae RAW) as the image data generating random variable with the target distribution D. = (V,Y) A different data model Proce generates a different view V = PPrroo (X (AwW)) of the same underlying raw sensor data generating random variable X Raw, resulting in the dataset drift
The raw inputs RRA are then further R.C...W, in our case the measurement hardware like a camera
DP= (V,Y)
This characterization of dataset drift is closely related to the concept of distributional robustness in the sense of Huber where "the shape of the true undcrlying distribution deviates slightly from the assumed model' [04. In the distributional robustness literature it is typically assumed that somelhing changes from D, to Di, however without exactly specifying what changes. This makes it unclear what is actually being compared, whether the considered change is plausible and what factors contribute to the change. Data models offer a way out of this ambiguity through exact description of the data generating process. In practice, a common reason for dataset drift in images is a change in the camera types or settings, for example different acquisition microscopes across different lab sites a and that lead to drifted distributions D, f Dr. Anticipating and validating the robustness of machine learning model to these variations an exact and traceable way not just an engineering concern but also mandated by quality standards in many industries 105 107] Omissions to perform physically faithful robustness validations has, among other reasons, slowed or prevented the rollout of machine learning technology in impactful applications such as large-scale automated retinopathy screening or yield prediction
108.. machine learning melanoma detection [09 110
11 from drone cameras.
Hence, the calls for realistic robustness validation of image machine lcarning systems are not merely an excrcise in intellectual novelty but a matter of integrating machine learning research with real world infrastructures and performance expectations around its core ingredient: the data.
1.1 The status quo of dataset drift controls for images
How can one go about validating a machine learning model's performance under image dataset drift? The dominant empirical tochniques can broadly be categorized into augmentation and catalogue testing. Each approach has particular benefits and limitations (see Table for a conceptual comparison).
Augmentation testing involves the application of perturbations, for example Gaussian noise, to already in order to approximate the effect of datasct drift. Given a processed datasct
processed images 43
112 13
Note that the nomenclature around dataset drift is as heterogenous as the disciplines in which it is studied. See [03] for good discussion of cross-diaciplinary taxonomy. Here are concerned with dataset drift defined in Equation D that is changes in V that are induced by changes in Proce which some works also refer to as covariate shift more generally as distribution shift.
We write an uppercase letter A for real valued random variable and lowcrease letter a for its realization. bold uppercase letter A denotes random vector and bold lowercase letter a realization. For N realizations of the random vector A we write aN.- The state space of the random vector denoted by A= A=( (A(w) Ω).
3

---

Published in Transactions on Machine Learning Research (05/2023)
Data Models for Dataset Drift Controls
D Physical model
Gradient flow
Processing steps
Measurement Optics
Data model
P Proc Sharp- De- noising
Standard ML pipeline
Sensor
Black
De- mosaic
White balance
Color correction
Gamma correction
Standardize Augment
Dask
level
ening
Model
Data states
DRAW
VBL
UDM
WB
VCc
USH
uDN
UGC
Novel =RAW enabled
drift drift controls
Drift synthesis
2
Drift forensics
3
Drift optimization
Train
Test
Train
Adversarial search
Train
Test
Figure 1: Schematic illustration of an optical imaging pipeline, the data states and novel, raw-enabled drift controls. Data transitions through different representations. The measurement process yields metrologically accurate raw data &RAW, where the errors on each pixel are uncorrelated and unbiased. From the RAA sensor state data undergoes stages of image signal processing (ISP) Prooc the data model we consider here. Finally, the data is consumed by machine learning task model Taakk which outputs ÿ. Combining raw data with the standard machine lcarming pipeline and a differentiable data model Proce enables useful controls for datasel drift comprising ! drift synthesis (creation of physically faithiul drift test cases for model selection), drift forensics (precise specification of data environments that should be avoided for given task model), and 3 drift optimization (use of task model gradient to optimize data generating process).
this allows fast and easy generation of test samples. However, 114 point out that perturbations applied to an already processed image can produce drift artifacts that are unfaithful to the physics of camera processing. Results in optics further support the concern that noise obtained from a composite image processing pipeline, such as the data model block in Figure [ is distinct from noise added to an image that has already been processed on hardware [1b] [16]. For illustration, assume we carry out augmentation testing test the robustncss of the task model wrt. to the dataset drift (1) Let E~ Doisse be noise sample additively applied to the the view resulting in É. Doing so, the task models robustness is tested wrt. the distribution Po (V+3)-1 that might not approximate D, well. Since P is unknown, this is difficult to resolve. For physically faithful robustness testing we need to ensure that sample is an element of the image P [X' RAW of X RAW under P Accordingly, we define a physically faithful data point wrt. the dataset drift (1) as view e that satisfies E Proc [X' R. augmentation testing, the test samples are not restricted physically faithful data points wrt. any datasct drift, since e + { @ [Aaw]] might not hold truc for any data model.
A physically faithful alternative to augmentation testing is what we call catalogue testing. It involves the collection of datascts from different cameras which are then used as hold-out robustness validation datascts It does not allow for as flexible and fast in-silico simulation of test cases as augmentation testing because cataloguing requires expensive data collection after which the test cases are "locked -nn" Notwithstanding, catalogue testing comes with the appcaling guarantce that test samples conform to the processing physics of the different cameras they were gathered from, ensuring that only physically faithful data points are used for testing.
4g
119
4

---

Published in Transactions on Machine Learning Research (05/2023)
Augmentation testing Catalogue testing
Data models
Simulation of test samples Physically faithful test samples Differentiable data model
X
X
Table I: A conceptual comparison of different empirical approaches to dataset drift validation for machine learning task models. While augmentation testing allows flexible, ad-hoc synthesis test cases, they are, in contrast to catalogue testing, not guaranteed to be physically faithful. Pairing qualified raw data with explicit data models allows for flexible synthesis of physically faithful test cases. In addition, the differentiable data model opens up novel drift controls including drift forensics and drift adjustments.
However, the root of imput data variations the data model of images has received little attention in machine learning robustness research to date. While machine learning practitioners are acutely aware of the dependency between data generation and downstream machine learning model performance, as 75% of respondents to a recent study confirmed [20], data models are routinely treated as black-box in the robustness literature. This blind spot for explicit data models is particularly surprising since they are standard
practice other scientific communities, in particular optics and metrology 121 124 as well as advanced
industry applications, including microscopy 127 or autonomous vehicles 128
1.2 Our contributions
In this manuscript we outline how to alleviate this blind spot and obtain data models by connecting conventional machine learning with physical optics. Combining raw image data, differentiable data models Proc of image signal processing (TSP) and the standard machine learning pipeline enables us to go beyond what is possible with augmentation and catalogue testing. We obtain explicit, differentiable models of the data generating process for flexible, physically faithful drift controls for image machine learning models. Our core contributions are:
The combination of raw sensor data and data models enables three novel dataset drift controls:
① Drift synthesis: Controlled synthesis of physically faithful drift test cases for model selection and largeted generalization. This is demonstrated for classification and a regression task and compared to physically unfaithful alternatives (Section 5.1
2 Drift forensics: Given particular data model T Proee the gradient from the upstream task model Ta propagate to Proce thus enabling precise data forensics: precise specification of data generation environments that should be avoided for a given task model (Section 5.2
3 Drift optimization: Lastly, the gradient connection between data Prroe and task model Prask allows directly adjusting the data and data generating process themselves via backpropagation to aid the downstream task model learn better faster (Section 5.3).
We collected and publicly release two raw image datasets in the camera sensor state for data modelling. The raw datasets come with full annotations and processing variations for both a classification (Raw- Microscopy, 17,860 total samples) and regression (Raw-Drone, 10,412 total samples) task. The data can be downloaded from the anonymized record https://zenodo http:///o...oorreeeor//////53530 as well through the data loader integrated in our code base below.
We provide modular Py code for explicit and differentiaable data models + Task from raw camera sensor data. All code accessible at htttp:://iith//
1.3 Scope, practical fit and limitation of the proposed methods
The systems infrastructure for optical imaging produced by large industry vendors such as ZEISS, Hamamatsu, Teledyyme-Photometriic, Andor, Yokogawa Porkin-Flmer allow raw sensor readouts. The applies
5

---

Published in Transactions on Machine Learning Research (05/2023)
to consumer-grade cameras by market-dominnting vendors such as Samsung Apple. Concurrently, ISP- processed data is predominantly used in practice both in the application domains considered here and machine learning. The reasons are often downstream workload dependencies. In most settings, data is acquired to be human readable for a specific task, for example diagnostics or environmental surveying. Variations in the ISPs, ween different vendors or acquisition sites, then lead to the drifts of Equation [ outlined above. This work targeted at the current imaging infrastructure that (i) makes widespread use of ISPs that lead to drift and (ii) simultaneously allows access to raw sensor readouts. For this setting, we propose data models that allow engineers to explore, emulate and control different data generating processes related to the ISP at low cost in a physically faithful way. In terms of practical benefits, data models can save time and money (drift synthesis) as additional acquisitions, on the order of days or even weeks, can be avoided. They also open up completely new applications for integrated data-model quality management (drift forensics, drift optimization) which are impossible without differentiable data models. To be clear, there are other important sources of drift, such as the sensor or optical components of the camera, which cannot be captured, yet, by our framework. For example, in the Raw-Mieroscopy dataset, factors such as the choice of filters for blue, green, and red channels, the point-spread function, and mechanical drift can influence the final image quality, Similarly, in the Raw-Drone dataset, the choice of lens, f-number, PSF circle diameter, ISO, and the gain applied to pixel values can affect the acquired images. These factors introduce variations contributing to dataset drift. The data models presented in this study aim to account for changes during the ISP. While extensions beyond this setting are opportune, raw-only, as sketched out in the drift optimization experiments, would require a shift in the dominant imaging workflows of the application domains considered here. In summary, the current data models offer rich utility to capture ISP as dominant source of data drift, but are limited to the ISP scope and require an extension to model additional factors outside that scope in a physically faithful manner.
2 Related work
While physically sound data models of images have to the best of our knowledge not yet found their way into the machine learning and dataset drift literature, they have been studied in other disciplines, in particular physical optics and metrology. Our ideas on data models and dataset drift controls we present in this manuscript are particularly indebted to the following works.
Data models for images I31 132 employ deep convolutional neural networks for modelling a raw image data processing which is optimized jointly with the task model. In contrast, employ parametric data model with tunable parameters that enables the modular drift forensics and synthesis presented later. [33 propose a differentiable image processing pipeline for the purpose of camera lens manufacturing. Their goal, however, is to optimize a physical component (lens) in the image acquisition process and no code or data is publicly available. Existing software packages that, provide low level image processing operations include Halide [34, Kornia [35] and the rawpy package [36] which can be integrated with our Python and PyTorch code. We should also take note that outside optical imaging there are arcas in machine learning and applied mathematics, in particular inverse problems such magnetic resonance imaging (MRI) or computed tomography, that make use of known operator learning [37 138 to incorporate the forward model in the optimization [39 or, as in the case of MRI, learn directly in the k-space [140.
Drift synthesis As detailed in Section [ the synthesis of realistic drift test cases for a task model in computer vision often done by applying augmentations directly to the input view ecc, e.g. a processod -jpeg or .pgg image. Hendryeks et al. 43 have done foundational work in this direction developing a practical, standardized benchmark. However, we explain in Section [.] there is no guarantee that noise g added to processed image a will be physically faithful, ie.. that & E Proc [xRAw]] This is problematic, nuances matter [11 for asscssing the cascading effects data models have on the task model Prakk downstream 120 122.
For the same reason, the use of generative models 47 like GANss has been limited for test data generation as they are known to hallucinate visible and less visible artifacts
[43 [44] Other approaches,
like the WILDS data catalogue 1. 140, build on manual curation of so called natural distribution shifts, or, like (8 on artificial worst case constructions. These are important tools for the study of dataset drifts, especially those that are created outside the camera image signal processing. Absent explicit, differentiable
6

---

Published in Transactions on Machine Learning Research (05/2023)
data models and raw sensor data, the shared limitation of catalogue approaches is that metrologically faithful drift synthesis is not possible and the data generating process cannot be granularly studied and manipulated.
Drift forensics Phan et al. [47 use a differentiable raw processing pipeline to propagate the gradient information back to the raw image. Similar to this work, the signal is used for adversarial search. However, Phan et al. optimize adversarial noise on a per-image basis in the raw space FRAW; whereas our work modifies the parameters of the data model Prroe itself in pursuit of harmful parameter configurations. The goal in this work is not simply to fool classifier, but to discover failure modes and susceptible parameters in the data model & that will have the most influence on the task model's performance.
Drift optimization An explicit and differentiable image processing data model allows joint optimization together with the lask model Prree- This has been done for radiology image data [148 [50 though the measurement process there is different and the focus lies on finding good sampling patterns. For optical data, a related strand of work is modelling inductive biascs in the image acquisition process which is explained and
contrasted to this work above 16 133
Raw image data Camera raw files contain the data captured by the camera sensors 121 In contrast to processed formats such as .peeg or .png, raw files contain the sensor data with minimal processing The processing of the raw data usually differs by camera manufacturer thus contributing to dataset drift. Existing raw data sets from the machine learning, computer vision and optics literature can be organized into two categories. First, datasets that are sometimes treated- - usually not by the creators but by users of the data- as raw data but which are in fact not raw. Examples for this category can be found for both modalities considered here LE 103 All of the preceding examples are processed and stored in formats including -jeeg, .tiff, svs, - -png, mmp and .mov. Second, datasets that are labelled raw data which are raw. In contrast to the labelled and precisely calibrated raw data presented here, existing raw datasets 167 are collected from various sources for image enhancement tasks without full specification of the measurement conditions or labels for classification or segmentation tasks.
115 151 152..
3 Preliminaries: a data model for images
Before proceeding with a description of the methods we use to obtain the data models Proce in this study, let us briefly review the distinction between raw data TRAW: processed image v and the mechanisms T P - C.H.W by which image data transitions between these states Image acquisition has traditionally been optimized for the human perception of a scene [22 [68]. Human eyes detect only the visible spectrum of electromagnetic radiation, hence imaging cameras in different application domains such medical imaging or remote sensing are usually calibrated to aid the human eye perform a downstream task. This process that gives rise to optical image data, which ultimately forms the backbone for any machine learning downstream, is rarely considered in the machine learning literature. Conversely, most research to date has been conducted on processed RGB image representations. The raw sensor image &RAW obtained from a camera differs substantially from the processed image that is used in conventional machine learning pipclines. The FRAW state appears like grey scale image with a grid structure (see Braw in Figure D. This grid is given by the Bayer color filter mosaic, which lies over sensors [21. The final RGB image U is the result of series of transformations applied to TRAW. For many steps in this process different possible algorithms exist. Starting from a single ZRAW, all those possible combinations can generate an exponential number of possible images that are slightly different in terms of colors, lighting and blur - variations that contribute to dataset drift. In Figure Ia conventional pipeline from FRAw to the final RGB image v is depicted. Here, common and core transformations are considered. Note that depending on the application context it is possible to reorder or add additional steps. The symbol c is used to denote the 1th transformation and in (view) for the output image of Φ; The first step of the pipeline is black level correction æBL. which removes any constant offsct. The image UBL is grey image with Bayer filter paitern. A demosaicing algorithm DMM is applied to construct the full RGB color image [69]. Given UDM, intensities are adjusted to obtain a neutrally illuminated image UWB through a white balance transformation T WWB- By considering color dependencies, a color correction transformation &cc is applied to balance hue and saturation of the image. Once lighting and colors are corrected, a sharpening algorithm 4SH is
We recommend
122 for good introduction the physics of digital optical imaging.
7

---

Published in Transactions on Machine Learning Research (05/2023)
applied to reduce image blurriness. This transformation can make the image appear more noisy. For this reason denoising algorithm DDx is applied afterwards 170 171.. Finally, gamma correction, &cc, adjusts the linearity of the pixel values. For a closed form description of these transformations see Section 4.2 Compression may also take place as an additional step. It is not considcred here as the input image size is already small. Furthermore, the effect of compression on downst task model performance has been thoroughly examined before 176 However, users of our code can add this step or reorder the sequence of steps in the modular processing object class per their use case needs
4 Methods
In order to obtain advanced data models for images, raw sensor data is roquired. In many industry domains such as microscopy, biomedicine, autonomous vehicles or remole sensing raw sensor data is processed at scale for machine vision tasks. Most existing digital camera systems on the market today, including consumer smartphones, can be configured to access the raw sensor measurements. Next, we explain how to obtain raw sensor data from existing optical hardware. We collected two datasets for two representative machine learning tasks. Both datasets are made available for free, public use at http:://zaandd..orr/reroc///////
4.1 Raw dataset acquisition
As public, scientifically calibrated and labelled raw data. is, to the best of our knowledge, currently not available, we acquired two raw datasets as part of this study: Raw-Microscopy and Raw-Drone. Raw- Microscopy consists of expert annotated blood smear microscope images. Raw-Drone comprises drone images with annotations of cars. Our motivation behind the acquisition of these particular datasets was threefold. First, we wanted to ensure that the acquired datasets provide good coverage of representative machine learning tasks, including classification (Raw-Microscopy) and regression (Raw-Drone). Second, we wanted to collect data on applications that, to our minds, are disposed towards positive welfare impact in today's world, including medicine (Raw-Microscopy)) and envirommental surveying (Raw-Drone). Third, we wanted to ensure the downstream machine learning application contexts are such where errors can be costly, here patient safety (Raw-Microscopy) and autonomous vehicles (Raw-Drone), hence necossitating extensive robustness and dataset drift controls. Since data collection is an expensive project in and of itself we did not aspire to provide extensive benchmark datasets for the respective applications, but to collect enough data to demonstrate the advanced data modelling and dataset drift controls that raw data enables.
In Appendix 1.4.1 we provide detailed information on the two datasets and the calibration setups of the acquisition process. Samples of both datasets can be inspected in Figure 2 and Appendix A.4.1 Full datasheet.
documentation following | is also available in Appendix A.4.2
An alternative approach could be to attempt reconstruct raw images from processed images 178 179. As we laid out earlier, when an image is captured by a digital camera, the sensor records raw image data in the form of an array of pixel values. This raw image data is usually processod by an ISP before being used in a downstream lask. The ISP performs various adjustments such as color correction, noise reduction, and sharpening to enhance the quality of the image. However, these adjustments the original raw image data and result in loss of information. Therefore, it is generally not possible to reconstruct the exact raw image data from the ISP processed images. While the processed images may look better to the human eye, they do not accurately represent the physical reality of the original scene. For example, a recent paper by Nam et al. [79] propose a content-aware metadata approach to sRGB-to-Rar Raw RGB do-rendering, but acknowledges that the resulting approximations are not perfectly accurate and still sulfer from limitations. Similarly, another study by Brooks et al. [78 present a method for recovering raw data from processed images, but also note that the approach is not able to recover all of the original data with perfect accuracy. These findings highlight the fundamental challenge of reconstructing raw data from processed images. Empirical approximations are possible but not exact, that is not physically faithful, and hence orthogonal to our goal here. However, one should note that these reconstruction approaches can offer
not physically faithful to
4See pipeline pipeline_torch.pee torch. py and pipeline sipeline_numyy-pp numpy-py our code.
8

---

Published in Transactions on Machine Learning Research (05/2023)
Microscopy
Drone
Prucc (RRAW)
PProe (RAW))
y
Procl (TRAW)
y
Proc (TRAW)
(LTT) pesiaf Anmmeeenr
Neutrophil band (NGB)
Figure Processed samples and labels of the two datasets, Raw-Microscopy (columns one to four) and Raw-Drone (columns five and eight), that were acquired for the dataset drift study presented here.
interesting value propositions outside the physically precise drift regime. The proposed technique by [78 "umprocesses' images and offers interesting gains during training, enabling a convolutional neural network with 14-38% lower error rates and 9-18x faster performance, while generalizing other sensors well. This approach can further be calibrated using joint learning of sampling and reconstruction, offering better reconstructions by adapting to image content, with an additional online fine-tuning strategy for enhanced results 179
4.2 Data models: Image signal processing &rroc
RH.W x The second ingredient to this study the data models of image processing. Let (X RAW, be the raw sensor data generating random variable classification and y {0, H.W. for segmentation. Let T R.CH.W + be the task model determined during training. The inputs that are given to the task model Prakk are the outputs of the data model $ Proc- We distinguish between the raw sensor image &RAW and a view v = Drroe (XRA)) of this image, where 4 Proc: R.W + HCH models the transformation steps applied to the raw sensor image during processing,
probability space (D,F,P), with y {0,1)* for
The objective in supervised machine learning is to learn task model Prakk &RCHW. within a fixed class of task models H that minimizes the expected loss wrt. the loss function f: x x [0,00), that is to find T Task such that
inf B[C Task V),Y)]
(2)
TTH is attained. Towards that goal, Task is determined during training such that the empirical error
N Σ L (PPTas(VN)
33)
is minimized over a sample S ((U1,,1)..., yN)) of views. Modelling the conventional machine learning setting begins with the image data generating random variable (V, VY = (#Proc (X RAw)), and the target distribution D, IP (V,Y)-!. Given dataset drift D. P Y)- D., as specified in Equation ( without data model have little recourse disentangle reasons for performance drops in Task To alleviate this underpecciiiatioon, an explicit data model is needed. We consider two such models in this study: a static model Ppice and parametrized model + para
Proce
In the following, we denote by FRAW [0, 10 H.W the normalized raw image, that is a grey scale image with a Bayer filter pattern normalized by 216 1, Le..
A
A 1,
PIAWW
with Anj
1,2+1
g2h.2w+1
92h+112w
b2h.
(4)
Ag
A Ag.u
where the values 12f+1.2w41. 1 1, 2w Q2t.=1.2wr99hh.2eee b2n.2w correspond to the values measured through the different sensors and normalized by 2 1.. We provide here precise description of the transformations that consider

---

Published in Transactions on Machine Learning Research (05/2023)
Data models
Used funetioos
bis,ame
RX
bis.ga
S2
bi,u,me
4
+
SE
F0.A.10 ma.-1.2 PA.
10.2..0
bi, Muge
4: 400
4
Ine Sume
MLU
me..g.a
2006
48 01
mag,uge
2
49
48
ma.,ine
4 10a
ma,sge
1234
511
mu,u,me
49 11L
ma,iiga
48
(a) Samples for both datasets, Raw-Microscopy and Raw-Drone, from all twelve static data models tat used for the drift synthesis experiments in Section 5.1 A version with higher resolution is omitted here to save space and can instead be found in Figure in the appendices.
(b) Abbreviations the twelve coufigurations of the static dala model + star. Proe used in the drift synthesis experiments.
Figure Samples for both datasets (a) and abbreviations of the twelve data models (b)
in our static model stat Proce
followed by description how convert this static model into a differentiable,
parametrized model + Proc
4.2.1 The static data model Pproc
Following common steps in ISP, the static data model is defined the composition
(5) mapping a raw sensor image to RGB image. We note that other data model variations, for example by reordering or adding steps, are feasible. The static data models allow the controlled synthesis of different, physically faithful views from the same underlying raw sensor data by mamually changing the configurations of the intermediate steps. Fixing the continuous features, but varying 4DM, 2sH and @D results in twelve different views for the configurations considered here. Samples for each of the twelve data models are provided in Figure 3a The individual functions of the composition Øplae can be found in Appendix AA.1.1. An overview of the data model configurations and their corresponding abbreviations can be found alongside processed samples in Figures 3a and 3b
stat PGC o LDN 4SH &cc @wB 4DM LBL.
4.2.2 The parametrized data model 4 para
Proc
For fixed raw sensor image, the parametrized data model Para maps from parameter space θ to RGB image. It is similar to the static data model with the notable difference that each processing step is differentiable wrt. its parameters 0. This allows for backpropagation of the gradient from the output of the task model Task through the data model Proce all the way back to the raw sensor image FRAW to perform drift forensics and drift adjustments. Hence, we aim to design a data model o p H.W.W θ R.HH. that is differentiable A ④ satisfying
$ for some choice of parameters stat and some fixed configuration of the stalic pipeline Prece- Using the individual functional components specified in Appendix A.1.2 we define for 0 (01...., 07) E θ the parametrizod processing model
Prac stat $ Para (-.0 1.0stat
(6)
stat
o para
[0. HJ..H.W θ + [0, 10,1)3.H.W (æRAW; 0)
(7)
by the composition v= (*eer (-,0)) o para (:, 0a) a (.,0g) par (,, 02) o WNE (.00) o *ew (.0z) a (:, (#RAW)
para
(8)
The operations used above are differentiable except for the clipping operation the GC that a.e.- diffcrentiable since the set {0, 1) of non-difforentiable points has mcasure zero. Assuming addition that P ((UDN)).h. (0,1)) = holds true for the entries of uDN results in a.e-diiferrnniable processing
5 Sa.e. stands almost everywhere
10

---

Published in Transactions on Machine Learning Research (05/2023)
model. We further say that $ Proc is differentiable, noting that this holds only a.e. under the aforementioned
assumption.
4.3 Task models 4 Task
Finally, two task models are employed in the experiments. For the classification task on the Raw-Microscopy dataset a 18-layer residual net (ResNet18) [SD was used as reference task model. To segment cars from the Baw-Drone dataset the convolutional neural network proposed in [8T (U-Net) was used. Both task models were trained using common data augmentations on processed views a of the image measurements to avoid naive robustness failures. A detailed description of the task models and their hyperparamotors is given in A.2
5
Applications
With data models, raw data and lask models in place we are now able to demonstrate the ad datasel. drift controls comprising I drift synthesis, 2 modular drift forensics and 3 drift optimization.
5.1 Drift synthesis
The static data model enables physically faithfiul synthesis of drift test cases: individual components of the data model can be swapped out, allowing the controlled creation of different, physically faithful processed views from one raw reference dataset. A typical use case of drift synthesis for machine learning researchers and practitioners is the prospective validation of their task model to drift from different camera devices, for example microscopes across different labs, without having to collect measurements from the different devices. This simulation can be done insilico as software because the hardware specific processing that takes place on optical measurement devices after the sensor reading is also insilico. Thus, the extraction of raw sensor readings from one device allows the emulation of different processing variations present on other devices. A typical workflow of this data synthesis starts with an engineer constructing a data model of interest, then passing raw measurements through it and finally getting emulated data to test how the downstream task model would fare on processing variations from different devices.
We provide twelve possible example data models in the following experiments. For each of the twelve data models laid out in Section 4.2] the task models were trained for 100 epochs on image data processed through the training data model. Hyperparameters were kept constant across all runs to isolate the effect of varying the data models. Then, dataset drift test cases were synthesized by processing the raw test data through the remaining cleven data models. The task models were then evaluated on test data from all twelve data models. All results that follow are reported as the mean with error bars over 5-fold cooss-validttion? The metrics used to evaluate the task models are accuracy for classification and loU for segmentation.
5.1.1 Physically faithful versus physically unfaithful robustness validation
The leukocyte classification model, as displayed in the left matrix of Figure 4 has critical drop for few configurations, suggesting that it relatively robust to processing induced dataset drift except for the (ma,,,me) configuration. Note that diagonal elements serve as reference corresponding to test data that was processed in the same way as the training data. The segmentation task model (left matrix in Figure displays more heterogeneous pattern with symmetries for certain combinations of data models, such as (bi, u, me/ga) and (me, s, me/ga), which are mutually destructive to the task model performance. The average performance of the task models drops from 0.82 to 0.8 between train and test data models for classification and from 0.71 to 0.65 for segmentation. That is the average change from train to test data environment calculated across all configurations for ISP as well as Common Corruptions. The results for individual components the data models can also be directly compared in Figures 4 and 5 For example, to understand how changes in the demosaicing algorithm affect the segmentation model, we can look at the left box in Figure 5 and focus on the column combinations 1-5-9, 2-6-10, 16610,, 3-7-11 and 4-8-12 where the demosaicing is variod but the other components of the data model stay fixed. Considering the training condition with the
You can find full description of task model hyperparameters and experimental setup in Appendix A.2
11

---

Published in Transactions on Machine Learning Research (05/2023)
Drift synthesis with patat
Microscopy
ISP
Common corruptions benchmark
0.84
0.05
E12
143
A31
223
1:0
bla,me
ACC
AM
UA4
0.75
0.78
bhamm
X20
Bla.ge
3
0.72
me
2:11
0.64
22
2
3
:
8
+
me
-035
me s.go
5
2
F
g
0.56
me makuuut
021
19
0.25
M
5
0.92
DE ga
messgge
E
3
0.15
0.48
Harrity
gosss Nelsee
allct
impuliee
centrest
brightent
satorate
elasti
issme
Iksme
Buggs
naneme
masgga
masuue
masipa
metkmm
me gg
misuue
me. ga
ppelines
0.03
0.06
0,18
Deviation
Standard Devistion
ma
[A
[A-8]
gueen
[A-8]
SLage
[A-
1o
[4
grean
Figure 4: 5-fold vross-validation results of the Raw-Microscopy drift synthesis experiments. Each cell contains the average accuracy with color coded border for the standard deviation. Task models were trained on the data models on the vertical axis and then tested on processed data as indicated the horizontal axis. Numbers 1-3 left to the vertical axis denote the ranking of task models according to their average accuracy across all test pipelines (respective corruptions). Stars denote the train pipeline under which the task model performed best on the respective test sipeline/ccoruption.. Full ranking results can be found in Tables 5 to 7 of Appendix A.3 Top-left: Varying the data model leads to mild performance drops except (ma,s,me) Diagonal L Proc PProe Top-Right: Comparison to the corruption benchmark at medium severity (level 3). The average performance drop is more than thirteen times higher compared to data model variations. First column is Prooe Procc Bottom: Visual inspection of worst case (globally worst scoring) train/tect /test pipelines.
mbi, data model using Bilinear demosaicing (row 1), the task model performance drops from 0.7 (columm 1) to 0.67 (column 5) IoU in response to Malvar2004 demosaicing and 0.66 (column 9) IoU when using Menon2007.
To demonstrate the limitation of post-hoc augmentation. compare the drift synthesis results to popular augmentation testing framework known as Common Corruptions Benchmark [3]. In machine learning practice, augmentation users often assume that applying a corruption, for example 'blur', to a processed image will emulate the noise from a real-world camera system, for example blur from the lens or the denoising component in the camera. However, this is not the case. should not come as a surprise given the composition of the optical data generating process (see Figure [ and Section 4.2), that is v u+ &Proe [ARAw]] might not hold true for any data model we explain in Section 1.1 This has also been empirically demonstrated in previous work T4 [6. Here we go one step further to show that physically unfaithful augmentation testing can lead to wrong conclusions in model selection.
We are only referring limitations relating robustness testing and model selection. Augmentations have important. empirically validated benefits in other applications such regularization during training or semi- and self-supervised learning.
12

---

Published in Transactions on Machine Learning Research (05/2023)
Drift synthesis with &rroc: Drone
ISP
Common corruptions benchmark
0.72
0.73
AUB
14S
XUB
bla,me
SG4
0.7
0.67
6:2
3:00
0.68
Waum
blamm
0.90
0.54
644
1
0.6G
Blacge
2
0.64
mas.mm
Taan pioline
0.62
3
na na...n
065
800
0
CC
800
OC7
OA?
MA4
0.58
maxu.go
miksgge
AG4
8:0
me
662
A34
0.0
s.ga
637
me s.go
3
0.27
640
647
sume
metsggo
0.17
0.48
Harrity
gosss neise
aloot
impuliee
Centrest
brightrens
satorate
elastic
bsd,me
Bedgl
t Maams
Bldgl
mas.ge
masapa
ma
mEaume
mp,gge
nau
ppeines
0.01
0.02
0.03
0.04
0.05
0.08
0.09
Standard Devistion
[A-S]
[A
grean
[A-0]
[A-8]
mB
[A-0]
greert
[A-e]
Mum
Figure 5: 5-fold cross-validation results of the Raw-Drone drift synthesis experiments. Each cell contains the average IoU with color coded border for the standard deviation. Task models were trained on the data model on the vertical axis and then tested processed data indicated on the horizontal axis. Numbers 1-3 left to the vertical axis denote the ranking of task models according to their average ToU across all test pipelines respective corruptions. Stars denote the train pipeline under which the task model performed best on the respective test sipeline/corrupttiinn
Full ranking results can be found in Tables and of Appendix A.3 Left: Varying the data model leads to mixed performance drops. Diagonal is PProe Proc- Right: Comparison to the corruption benchmark medium severity (level 3). The average performance drop is more than four times higher compared to data model variations. First column is &Proe -Prrec- Bottom: Visual inspection of worst case (globally worst scoring) train/test pipelines.
As we note in Appendix A.3 a direct apple-to-appl comparison is impossible due to the fundamental limitation of post-hoe corruptions' physical unfaithfulness. However, we make the comparison as plausible as possible by only including corruptions that can be related to the ISP data model. Others, such as Fog, Spatter, Motion, Snow, Frost were excludec In contrast to physically faithful test data, the performance drops under corruptions are more severe across the board: from 0.82 to 0.55 for classification and from 0.71 to 0.49 for segmentation" This more than thirteen and four times as much as for the physically faithful drifts synthesized with the data models considered here. We see similar gaps when considering the best models. For example, the best performing microscopy data-lask-moddl combo selected across all test ISPs (mma,s,me) with 0.83 average accuracy) has more than 20 percentage points gap compared to Common Corruptions (0.62 average accuracy). For the segmentation task we make a similar observation where the best performing drone data-task-modd combo selected across all test ISPs ((ma,u.g)) with 0.68 average IoU) has more than 10 percentage points gap compared to Common Corruptions (0.55 average ToU) (see Appendix A.3 for full tables.) The qualitative difference between physically faithful drift test cases and augmentation testing can also be appreciated in the samples of the bottom rows of Figures and For each task we display sample
"A comparative overview of included and excluded corruptions be found in Figure 1 Appendix A.3
9 Results additional severity levels for the common corruptions can be found in Appendix
13

---

Published in Transactions on Machine Learning Research (05/2023)
from the drift test configuration with the worst case performance drop between train and test data conditions. We show the sample viewed from training data model (A); the test data model (B), and the difference between both ([A-B() along the red, green and blue channel. For both tasks, the drift artifacts ([A-B)) are more localized than the artifacts obtained from augmentation testing. This makes sense, as changes in the composition of the test data models Proe maintain the physical faithfulness of the remaining data model, whereas augmentation testing spreads noise globally across all pixels which not guaranteed to be physically faithful.
5.1.2 Implications for model selection
Similarly, the conclusions for model selection diverge depending on whether physically faithful data or corruptions are used. In terms of the average performance across all test conditions, none of the top-3 training data _ [T0] overlap between ISP and common corruptions on the classification task. For segmentation, only one of the training data models (bi,s.ga) overlaps in the top-3 under ISP and common corruptions. Similarly, the training data models under which task models perform best in individual testing conditions vary widely between ISP and common corruptions, both for classification and segmentation. Why does physical faithfulness matter in dataset drift testing? A test result is only as reliable as its constituting parts. If we are to rely on robustness test results to decide whether to use a task model in a certain data environment or not, we need to ensure the test cases represent real-world data models. If the test cases are not physically faithful, the results based on them are of limited use to make decisions.
5.1.3 Data models and targeted generalization
Recent advances in learning theory by Krikamol Muandet conjecture the impossibility to design rational learning algorithms that have the ability to successfully learn across heterogeneous environments [66 Explicit data models allow us to rethink the problem of generalization in similar vein. With data models it is possible to i) precisely specify individual environments and ii) observe what combinations of environments and task model play together nicely. Rather than selecting task models with best average performance across all heterogeneous lest, enviromments, we can serve the task model with the right data model depending on which environment it is deployed in. When we analyze the columns of the matrices in Figures | and 5 observe under what training data model (or 'environments' Muandet's language) the task model performs best in which testing environment. These configurations are marked by a star (x). For example, in the case of classification we can observe that for task model to perform well in (bi,u,me) and (bi,u.ga) test data environments, the (ma,u,ga) training data environment is best (left matrix, Figure ( However, for the segmentation task, to perform best in the same testing data environments, the (bi,u,me) training data environment is preferable (left matrix, Figure 5
we can
5.1.4 Use cases and limitations of drift synthesis
The most immeddate use cases for drift synthesis is physically faithful, prospective validation without measurement. In this scenario an engineer will have a task model as well as reference raw measurements. She would then construct data models of interest, for example for two different microscopes across laboratory sites 8 and t. She would then pipe the reference measurements through data models s and t to obtain two different datasets and test the task models on them. She could observe the effect of the processing in lab site from a computer without ever having to take expensive measurements on site. Building up catalogue of data models, as demonstrated in the experiments, then further allows to perform model selection or targeted generalization management where the task model is paired with suitable data models during deployment. All these applications presuppose access to raw data as well as knowledge of the data model specification so that they can be constructed accordingly in software.
5.2 Drift forensics
Clear and precise specification of the limitations of use is a mandated requirement for many products that can potentially contain machine learning components, such as software as a medical device [05] 106 or
Denoted by the numbers 1-3 alongside the rows of the matrices Figures
14

---

Published in Transactions on Machine Learning Research (05/2023)
2
Drift forensics with VoAl V- V1 L(V, Y)
(a)
5,000
peanicur wuRch spa0
80
2500
5,000
60-
grena corvec shirpreng 18he
0.673
1600
grassinn
6.000
40
4.927
6000
4.000
20
134
10"4
10
10-4
repularaatoe v
10"1
10
10
10
10
Spplrrattiss woight
10-1
10-15
10-6
200
den
reference
10
10-3
10
10
5
dovic
10'-5
10"
Figure 6: (a): Test accuracy on the Raw-Microscopy test set after 20 epochs of adversarial search in the data model for varying regularization weight parameters X. The individual plots depiet the various pipeline parameter selections (left plot). Plot showing f--norm (of processed images between the adversarially trained क para and the default ( versus attained accuracy of the task model (right plot). The metries are evaluated on the test set after 20 epochs of advesariial optimization for varying regularization weight, parameter X. (c): Same for Raw-Drone. The individual plots depict the various data model parameter selections. A lower regularization results in a bigger search space for advensarial optimization. Forensics loss refers to the binary cross entropy and Dice loss used as the optimization objective for the segmentation task model. (b): Processed samples from the drift forensics after 20 epochs with varying regularization weights A.
autonomous vehicles 182. Without knowledge and control over the data acquisition process in practice this requirement cannot be met. An explicit, differentiable data model paired with raw data offers a viable solution to this problem. &prce enables the analysis of the task model's susceptibility to dataset drift in an interpretable manner using adversarial search. Related work, such 147 also uses differentiable raw processing pipeline to propagate the gradient information ion back to the raw image. There, however, the signal is used in a classical adversarial sctup, to optimize adversarial noise on a per-image basis. In our work here, gradient updates are not applied to individual images, but to the data model parameters. The goal of such an analysis is to identify the parameter configurations of the data model under which the task model should not be operated. The resulting adjustments correspond to plausible changes which reflect changes in data model, for example due to changing camera ISPs. In order to limit the parameter ranges, we chose an explicit constraint in the RGB space.
minimize Oce
\ - - LVV,),
where V + para RAW 0) are the RGB images obtained from the original data model and V o para
Proc RAW. 0) are the RGB images obtained from adversarial search on the data model parameters. Equation 9 maximizes the classification loss under a relaxed fg-constraint controlled by the hyperparameter 2 O. This procedure yields data model parameters that deteriorate the task model performance while keeping the measured distortion minimal and the within constraints of physical faithfulness. All of the pipeline's parameters are optimized jointly to search for a task model's overall data model related weaknesses. Targeting select paramcters is also possible and provides insight into paramcter's effect the task modcl's performance.
5.2.1 Sensitivity to data models
The left plot in block (a) of Figure 6 shows scnsitivitics of the classification task model to changes in the data model parameters. With increased relaxation of the lyrreularizztton, the accuracy declines exposing configurations under which the task model deteriorates. As to be expected, the setting allowing for all
15

---

Published in Transactions on Machine Learning Research (05/2023)
parameters to be altered shows the biggest effect on the resulting performance. Individually, changes in the black level configuration &B and the denoising parameters DN pose the greatest risk for task model performance under a relaxed regularization weight. In contrast to the classification model the performance drops for the sogmentation model are less severe (top plot in block (c) of Figure Gp. We hypothesize that this is because classification problems are inherently discontinuous while inverse problems inherently allow for more stable solutions 183 thus being less susceptible to instabilities.
5.2.2 Sensitivity in relation to magnitude
For comparison, the right plot in block (a) of Figure i shows the regularization weight À against the resulting l2. Interestingly, a higher norm in the resulting RGB images does not directly translate to the most severe performance degradation of the task model. At & 10-5, changes in the Gaussian blur parameters induce a norm almost twice as large as the changes in the black level parameters. However, the corresponding drop in accuracy caused by Gaussian blur is around one third less relative to black level. Similarly, at l2 10-5, the sharpening filter parameters incur norm but do not lead to accuracy drops of the lask model. This underscores the importance of precise data models for dataset drift validation. Physically faithful yet small changes, as visible in the samples in bottom row of Figure G in processed images can have larger impact on the performance than large changes.
5.2.3 Use cases and limitations of drift forensics
A practical use-case of drift forensics looks follows: party s develops and trains a model and then licenses it to party t for use. Party wants to know what the data conditions are under which the lask model performs well and under which conditions it should not be used. Party runs drift forensics and provides party t with a forensic signature, as seen in Figure f detailing which parameters in the data model can be changed and which should not be touched to maintain task model performance. Party t can use this information to calibrate their data processing and knows which data settings to avoid for the specific task model. As with the other drift controls, access to the raw data as well as data models is required to perform drift forensics.
5.3 Drift optimization
In the previous two experiments we demonstrated how raw data and a differentiable data model can be used to identify and then modularly test for unfavorable data models that should be avoided during deployment of the machine lcarning task model. The same mechanics can also be exploited to optimize the data itself, effectively creating a beneficial drift. In the drift optimization setting, the gradient from the task model Pask is propagated into the data model Proc to jointly optimize both of them. In the learned setting, the data model parameters are jointly optimized with the task model parameters. In the frozen setting, only the task model parameters are optimized and the data model parameters are kept fixed
5.3.1 Convergence and stability
In the left column (a) of Figure 7 these two scenarios are compared. The learned data model creates a drift that improves stability of the learning trajectory. This is indicated by the blue line which displays the validation accuracy against optimization steps for the first half of training (step 1439 corresponds epoch 60). It exceeds that of the frozen data model (red line) by up to 25 percentage points in accuracy at a lower variance. For the segmentation task (bottom row Figure 7 the stabilization effect cannot be observed. This could be due to the low resolution of the problem itself as the drift optimization may not have a large effect on enhancing the solid blocks of cars in the raw data. Other evidence further suggests that inverse problems are inherently less unstable [83]. The results of the convergence and stability behavior under the different settings can also be found as a tabular summary in Table
'he initialization of 4 peece (both frozen and learned) is standard values which he found in Appendix A.1 well as pipeline ieteiiitttna torch. py of the code.
16

---

Published in Transactions on Machine Learning Research (05/2023)
Drift adjustments with $ Prane
(a) Low intensity (0.001) Xnaw with Proo
(b) High intensity (1.0) Xnaw with 2
Frowen
Learned
Validation metric
Frozen
Learned
Validation metric
Raw comparison
Drone Microsoopy
6:5-
100
1090
9:30
1000
000
2
2000
20.0 6.00
Figure 7: Low (a) and high (b) intensity images processed by frozen and learned pipeline. This type of drift optimization would not be possible with processed data. The plots columns three and six display the mean of validation metrics over five cross validation runs. Column seven shows additional results on raw data for comparison. Error bars are reported as one standard deviation. Optimization step 1439 and 915 correspond epoch 60 into training.
5.3.2 Helpful artifacts
In fact, the processod image from a learned data model with optimized drift (sco learned column in block (a) of Figure for an example) can contain visible artifacts that aid stability and generalization vis-a-vis the image from the frozen baseline data model which, arguably, looks cleaner to the human eye. A possible explanation for the improved learning trajectory could be that varying optimized drift automatically generates samples akin to data augmentation. Such uses could further be explored in scarce data settings like fine tuning, semi-supervised or few-shot learning. Having gradient access to the data model thus offers the opportunity to optimize data generation itself for a given machine learning task. learned data models are to be applied in real- world applications, it thus appears likely that a tradcoff has be made between human perceived visual quality and artifacts that can be helpful to the task model.
Similar outcomes for stability and artifacts also be observed for the reverse situation (high intensity 1.0 #rAw) in the right column (b) of Figure
5.3.3 Raw and data models
We demonstrated how parametrized data models can be used to optimized drift under data model constraints. Going beyond physically faithful drift controls, an interesting extension to drift optimization is training directly on raw data to optimize task model performance. While in this manuscript we are concerned with providing building blocks to emulate optical data models used in practice, training dircctly on raw opens up the possibility to learn purely machine-oplinizzed optical data processing free of existing data model constraints. In the last column of Figure 7 an optimization directly on raw data is displayed for each task. Results are reported across threefold cross-validation with error bars of one standard deviation. Like in the other experiments the task model parameters are tuned well (see task model training details in Appendix A.2). For classification, a performance similar to the learned setting is achieved with more volatile optimization trajectory. For the segmentation task, the performance is not on par with either the learned or frozen setting, but appears plausible that this gap can be substantially reduced with further finetuning. Learning directly on raw data thus appears as a promising direction for data model-free machine vision.
5.3.4
Use cases and limitations of drift optimization
Drift optimization can be used to squeeze out performance from a task model by creating drift that is helpful. A common use case is the adjustment of imaging pipelines, such as microscopes, that have traditionally been optimized for human end users, for example medical staff, which are increasingly being used in conjunction with automated machine learning models, for example for cell detection. By adjusting the parameters in
17

---

Published in Transactions on Machine Learning Research (05/2023)
Microscopy Average accuracy
Drone
Average loU
Learned (low)
0.75 0.09
0.59 + 0.05
Frozen (low)
0.54 = 0.21
0.59 i 0.05
Learned (high)
0.78 = 0.08
0.74 + 0.04
Frozen (high)
0.67 = 0.14
0.71 + 0.05
Direct raw
0.75 # 0.07
0.60 a 0.07
Table 2: Tabular summary of the drift optimization results. The average accuracy and standard deviations over vross-validation runs and training steps are displayed, summarizing both the stability and converge trajectory for each sctting.
the data models of existing optical laboratory infrastructure performance gains can be achieved. Due to the improved convergence and stability of the optimization trajectory can also potentially be used in situotions where compute is expensive or scarce altogther. However, these benefits do not hold across all tasks, as we in the case of the segmentation model, and for cases where no data models are present, such as novel custom optical hardware, learning directly raw data offers promising exctension. This work targets imaging infrastructure that uses ISPs causing data drift while also allowing access to raw sensor readouts. The proposed data models enable engineers to emulate and control different data generating processes related to ISPs in a cost-cffective, physically faithful manner. These models save time and money and enable new applications for data-model quality management. However, they only capture ISP-related drifts and require extensions to model other factors. The ultimate goal could be to train on RAW data, with the current. pipeline serving as an interim solution until RAW data becomes more widely available.
6 Discussion
The main message we hope to convey in this manuscript is that black-box data models for images neither have to be the norm in machine learning research nor in engineering. Leveraging established knowledge from physical optics enables us to push the modelling goalpost further towards machine learning's core ingredient: the data. Paired with raw data, precise differentiable data models for images allow for advanced controls of dataset drift, common and far reaching challenge across many machine learning disciplines. Applications beyond nobustnnss validation in areas of machine learning that are also held back by black-box data, such as federated learning and formal model certifications, appear opportune, too.
Drift synthesis allows the crcation of physically faithful of drift test cases. In contrast to augmentation testing, the performance drops for physically faithful test cases are less severe across the board for both uses cases our experiments, changing the conclusions we arrive at for model selection and enabling new ways to think about generalization with targeted, data model specific deployment of task models. A plausible practical application scenario of drift synthesis for machine learning researchers and practitioners is the prospective validation of their task model to drift from different camera devices, for example microscopes across different lab sites or autonomous vehicles, without having to collect measurements from the different devices. Drift synthesis could also be interesting for other application domains that rely on data synthesis (semi- SS
and self-supervised learning 91
02 or on precise data models (aleatoric uncertainty quantification 72
dustof-distribution detection 34 87) While we cross-validated substantial number of data model
variations in our experiments, it should be noted that. further variations, for example by reordering or adding steps, are possible. Furthermore, it should not be overlooked that dataset drift can also be caused by factors outside the ISP data model, for example the optical components of camera. Our current data models are not yet capable of capturing factors that go beyond the ISP. Integrating work from lens manufacturing 183 to expand the reach explicit data models offers a promising next step for drift synthesis.
Drift forensics allow the precise specification of data model limitations of use for a given machine learning task model. Data models under which the task model should not be operated can be identified by gradient search and then documented. In our demonstration, the setting allowing for all parameters to be altered shows the
18

---

Published in Transactions on Machine Learning Research (05/2023)
biggest effect the resulting performance. Individually, changes in the black level configuration and the denoising parameters pose the greatest risk for performance of the task model at hand. Interestingly, a higher norm in the resulting RGB images does not directly translate to the most severe performance degradation of the task model. This underscores the importance of precise data models for dataset drift validation, In practice, clear specification of the limitations of use is a mandated requirement for many products that can potentially contain machine learning components, such as software as a medical device [055 T0G or autonomous vehicles [82. Drift forensics with explicit data models can help to utilize the precision of machine learning and data engineering to satisfy such regulatory constraints. Explicit data models combined with gradient search may also be interesting to explore in areas such as formal model verification 55 71] to obtain tighter error bounds. Other constraints beyond bg are feasible, depending on the particular use case to be analyzed, and can be plugged into our code [2
We also showed how differentiable data models can be used for drift optimization where the data generating process is jointly optimized with the task model parameters. It leads to improved stability of the learning trajectory on the classification task in both low and high intensity measurements. Interestingly, the processed image from learned data model can contain visible artifacts that aid stability and generalization vis-a-vis the image from the frozen bascline data model which arguably looks cleaner to the human eye. In practice, the extension of the gradient connection from the task model Trakk to the data model Proc enables the extension of machine learning right into the data generating process. Thus, data generation itself can be optimized to best suit the task model at hand. Furthermore, the stabilization effect could prove useful for learning problems where training is costly and speedup precious (for example large models or large datasets). This capacity could also be exploited in other areas that deal with heterogenous training or deployment, environments, such different clients in federated learning [7 or domain adaptation techniques [M4 However, the above drift adjustment benefits could only be observed for the classification task, not the regression task, possibly due to the low resolution of the segmentation problem. How far we can push the gradient into the real world is an interesting future direction for data modelling. Including more parts of the data acquisition hardware into the data model and consequently the machine learning optimization pipeline appears leasible [185] and represents an important next step in aligning machine learning with real world data infrastructures.
Finally, raw data, which is already routinely used in optical industries 125 130 for representative machine learning tasks has to become more accessible to researchers to align robustness research with physically faithful data models and infrastructures. While most optical imaging devices support the extraction of raw data and this procodure is well established in industry and physics, data collection procedures for machine learning robustness research still have to catch up in order to make raw datasets and their benefits more widely available. Norms around established benchmarking datasets of processod images, such as CIFAR or ImageNet, can slow down this progress. To that end, we collected and publicly release two raw image datasets in the camera sensor state. The assumptions with respect to the practicality of the procedures we propose here are mild in our eyes. Raw subsets of data could be stored and then pulled in-code from cloud storage, as demonstrated in the code that we provide, for the purposes of drift synthesis or drift forensics. Learned data models obtained from drift adjustments could be calibrated directly on hardware such that the bandwidth requirements would not change compared to current image acquisition and transmission. Better APIs to optical hardware would allow more rescarchers and industrics to make their raw data accessible and service a culture of data modelling that can help overcome the limitations of machine learning in the pure task model regime.
Use of Personal Data and Human Subjects The microscopy slides were purchased from a commercial lab vendor (J. Lieder GmbH & Co. KG, Ludwigsburg/ Settiigsburg/Germaayy /Germny) who attained consent. The drone dataset does not directly relate to people. Instances with potential PITs such as faces or license plates were removed. Full datasheet documentation following [ can be found in Appendix A.4.2
Negative Societal Impact Machine learning risk management, such as the drift controls, can make ML deployment possible and safer. More deployment translates to increases in automation. A net rink-benefit.
adg_axxv_aux____e__ 12 Argument args.adv sup_aaux__aaa lov_aux_losse loss in train.py
19

---

Published in Transactions on Machine Learning Research (05/2023)
analysis of automation is beyond the scope of this mamuscript. What do know is that steel can be cast into ploughs and swords. We are against the use of our findings for the latter purpose.
20

---

Published in Transactions on Machine Learning Research (05/2023)
References
[1] CL Wilson and MD Garris, Handprinted character database. technical report special database 1. Technical report, National Institute of Standards and Technology, 1990. 2
22] MD Garris and RA Wilkinson. Handwritten segmented characters database. technical report special database 3. Technical report, National Institute of Standards and Technology, 1992.
[3] Michael Garris. Design, collection, and analysis of handwriting sample image databases. (31), 1994-08-10 1994.
URL https::/tssaps..ntte htt.://owwbbbcccco/eeeeeeeeeeeee
[4] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-basee learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998. 2
[5] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009. 2
[6] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. volume 115, page 211-252, USA, dec 2015. Kluwer Academic Publishers. doi: 10.1007/ -11263-015-0816-y. URL httts:://oi..
1007/s11263-0155 -11-102666--1---1--111-
[7] Alex Krizhevsky, Ilya Sulskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Commun. ACM, 60(6))84 90, may 2017. ISSN 0001-0782.
doi: 10.1145/30653888 URL
https://dbi.orga: 0ttp//00.115//366600 1145/3065386
[8] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[9] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint orx; 1710. 10196, 2017. 2
[10] Hamed Valizadegan, Miguel J. S. Martinho, Laurent S. Wilkens, Jon M. Jenkins, Jeffrey C. Smith, Douglas A. Caldwell, Joseph D. Twicken, Pedro C. L. Gerum, Nikash Walia, Kaylie Hausknecht, Noa Y. Lubin, Stephen T. Bryson, and Nikunj C. Oza. ExoMiner: A highly accurate and explainable deep learning classifier that validates 301 new exoplanets. The Astrophossical Journal, 926(2):120, feb 2022. doi: 10.3847/ 00.3347/1558-455 11111-55883775551 /3499993999 URL https://doi.cc otg/////////// 10. 00l.384711533 4357/ac4399
[11] Thomas J Fuchs and Joachin M Buhmaan. Computational pathology: challenges and promises for tissue analysis. Computerized Medical Imaging and Graphics, 35(7-8):515-555 530, 2011. 2
[12] T Terwilliger and MJBCJ Abdul-Hay. Acute lymphoblastic lenkemia: a comprehensive review and 2017 update. Blood cancer journal, 76): 2017.
[13] Christian Matek, Simone Schwarz, Karsten Spiekermann, and Carsten Marr. Human-level recognition of blast cells in acute myeloid leukaemia with convolutional neural networks. Nature Machine Intelligencr, ((11):538 544,
2019. !
[14] Qiwei Wang, Shusheng Bi, Minglei Sun, Yuliang Wang, Di Wang, and Shaobao Yang. Deep learning approach to peripheral leukocyte recognition. PloS one, 14(6):e0218808, 2019. 2
[15] Daisuke Komura and Shumpei Ishikawa. Machine learning methods for histopathological image analysis. Computational and Structural Biotechnology Journal, 16:34 42, 2018. ISSN 2001-0370. doi: https://doo.org// 100 btb.:j.csbj.2018.01111 1..00. URL
[16] Miriam Hägele, Philipp Seegerer, Sebastian Lapuschkin, Michael Bockmayr, Wojeiech Samek, Frederick Klauschen, Klaus-Robert Müller, and Alexander Binder. Resolving challenges in deep learning-based analyses of
histopathological images using explanation methods. Scientific Reports, 10(1):1-12, 2020. 2
[17] Maclej Wojtkowski, Tomasz Bajraszewski, Iwona GorezyA,ska, Piotr Targowski, Andrzej Kowalezyk, Wojeiech Wasilewski, and CzesA,a Radzewicz. Ophthalmic imaging by spectral optical coherence tomography. American Journal of Ophthalmology, 138(3)412 419, 2001. ISSN 0002-9394. doi: https:: /doi./roiiooor//eoo wtt.d/i..or./..1........ 111.0/0///.....20........ URL https::/wwet httpp::/www.scieeceeiiicce www.ssceweecnnccccoomm www.iiarrttttccceeeeeeemmemeeeee wwt.ccccc/ccciiie wttpe///aaariieee//////e////////
[18] Daniel Shu Wei Ting, Louis R Pasquale, Lily Peng, John Peter Campbell, Aaron Y Lee, Rajiv Raman, Gavin Siew Wei Tan, Leopold Schmetterer, Pearse Keane, and Tien Yin Wong. Artificial intelligence and deep learning in ophtthalmology. British Journal of Ophtthabmology, 103(2):167-17,, 2019. ISSN 0007-1161. doi: 10.1136/bb wwwpbbphhhhoool201ss. 2011-1m111220111111......3 URL httpps://jjo.. bmj. http://cccooomtttee com//contaa///////////5
[19] Yan Tong, Wei Lu, Yue Yu, and Yin Shen. Application of machine learning in ophthalmic imaging modalities.
Faye and Vision, 7(1):1-15, 2020. 2
21

---

Published in Transactions on Machine Learning Research (05/2023)
[20] MT Makler, CJ Palmer, and AL Ager. A review of practical techniques for the diagnosis of malaria. Annals of Tropical Medicine and Parasitology, (2(4):419 433, 1998. 2
[21]
Mahdiih Poostchi, Kamolrat Silamut, Richard J. Maude, Stefan Jaeger, and George Thoma. Image analy-
sis and machine learning for detecting malaria. Translational Research, 194:36 55, 2018. ISSN 1931-5244. doi: https://doi.org//t 10. 1rs.1///.tsll.211..2222
www.:/.ceeneeiiec....m http://ccccaaaaaaaeccccccccc//ea cttp////cccccmm//aaacccca/////// 5193152441730333X In-Depth Review: Diagnostic Medical Imaging.
[22] KM Fuhad, Jannat Ferdousey Tuba, Md Sarker, Rabiul Ali, Sifat Momen, Nabeel Mohammed, and Tanzilur Rahman. Deep learning based automatic malaria parasite detection from blood smear and its smartphone based application. Diagnostics, 10(5):329, 2020.
[23] Rose Nakasi, Ernest Mwebaze, Aminah Zawedde, Jeremy Tusubira, Benjamin Akera, and Gilbert Maiga, A new approach for microscopic diagnosis of malaria parasites in thick blood smears using pre-trained deep learning models. SN Applied Sciences, 2(7):1 7, 2020. 2
[24]
Markus Reichstein, Gustau Camps Valls, Bjorn Stevens, Martin Jung, Joachim Denzler, Nuno Carvalhais, et al. Deep learning and process understanding for data-driven earth system science. Nature, 566(7743):195-204) 2019.
[25]
David Rolnick, Priya L. Donti, Lynn H. Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, An- drew Slavin Ross, Nikola Ailojevie-Dupont Natasha Jaques, Anna Waldman-Brown, Alexandra Sasha Luccioni, Tegan Maharaj, Evan D. Sherwin, S. Karthik Mukkavilli, Konrad P. Kording, Carla P. Gomes, Andrew Y. Ng, Demis Hassabis, John C. Platt, Felix Creutzig, Jennifer Chayes, and Yoshua Bengio. Tackling climate change with machine learning. ACM Comput. Sure., 55(2), feb 2022. ISSN 0360-0300. doi: 10.1145/348514 URL https://do..org/// 10. 1145/3485128
[26] Qiangqiang Yuan, Huanfeng Shen, Tongwen Li, Zhiwei Li, Shuwen Li, Yun Jiang, Hongzhang Xu, Weiwei Tan, Qianqian Yang, Jiwen Wang, Jianhao Gao, and Liangpei Zhang. Deep learning in environmental remote sensing: Achievements and challenges. Remote Sensing of Environment, 241:111716,, 2020. ISSN 0034-4257. doi: lutlps//doi..rg $0003425720300867
[27] Jolm Quinn, Vanessa
and Lakshminarayan Subramanian. Computational sustainability and artificial intelligence in the developing world. AT Magazine, 35(3):36 47, Sep. 2014. doi: (0.1609/aiman hatp//aamaaa........... v3313. 2529. URL htttps://jjs..ae aaai. wrt..ooog//////eee httg//idx...h//iattee hhtp://aamaaainnararrieee.
2529
[28] Vittorio Mazzia, Lorenzo Comba, Aleem Khaliq, Marcello Chiaberge, and Paolo Gay. Uav and machine learning based refinement of satellite-rriven vegetation index for precision agriculture. Sensors, 20(9), 2020. ISSN 1424-8220. doi: 00.3390/s20092500 URL hattpps:/www.... www..mmpi..oo wwt.lococcccc/o/ 0124-8220/20/9/2530
[29] Abhinav Sharma, Arpit Jain, Prateek Gupta, and Vinay Chowdary. Machine learning applications for precision agriculture: A comprehensive review. IEEE Access, 9:1843 4873, 2021. doi: 10.1109/ ACCESS 201ESSS.0000004411555
[30] Odei Carcia-Garin, Toni MonleA"n-Getino,, Pere LÄOpez-Brosa, AsunciA'n Borrell, Alex Aguilar, Ricardo Borja-Robalino, Luis Cardona, and Morgana Vighi. Automatic detection and quantification of floating marine macro-litted litter in aerial images: Introducing a novel deep learning approach connected to a web application in r. Environmental Pollution, 273:116490, 2021. ISSN 0269-7491. doi: httpps//doii.rrg. 110 1016/j.envpol.2000 2021. 116499.. URL http:///wwwweeeeeeeeee.t.cooo/// http:///wcccceeaaree////////////
2
[31] Ferda Ofli, Patrick Meier, Muhammad Imran, Carlos Castillo, Devis Tuia, Nicolas Rey, Julien Briant, Pauline Millet, Friedrich Reinhard, Matthew Parkan, et al. Combining human computing and machine learning to make sense big (aerial) data for disaster response. Big data, 4(1):17 59, 2016.
[32] Monique M Kuglitsch, Ivanka Pelivan, Serena Ceola, Mythili Menon, and Elena Xoplaki. Facilitating adoption of ai in natural disaster management through collaboration. Nature communications, 13(1):1 3, 2022. 2
[33] Christian Szegedy, Wojeiech Zaremba, Πlya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1812.61 61999, 2013. 2
[34] Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojeiech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 2021. [8]
[35] Adarsh Subbaswany, Roy Adams, and Suchi Saria. Evaluating model robustness and stability to dataset shift. In Arindum Banerjve and Kenji Fukumizu,, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 2611 2611-2619. PMLR, 13-15 Apr 2021. URI. httpa: httpa://prooceeiing.. mlr. wwwppeewaaaa00pp http:///////ubaaaan2aa.2.. html 2
22

---

Published in Transactions on Machine Learning Research (05/2023)
[36] Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Monlavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10(1):1-8, 2019.
[37] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In H. Wallach, H. Tarochelle, A. Beygelz- imer, F. d' d'Alché-Bue, F. Fox, and R. Garnett, editors, Advances in Newral Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL /attps://proceedding neurips cc/papper/209///1lee .df
[38] Robert Geirhos, HeeniH Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Mechine Intelligence, 2(11):665-673,
2020.
[39 Emma Pierson, David M Cutler, Jure Leskovee, Sendhil Mullainathan, and Ziad Obermeyer. An algorithmic approach to reducing unexplained pain disparilies in underserved populations. Nature Medicine, 27(1):136 140, 2021.
[40] Judy Wawira Gichoya, Imon Banerjee, Anauth Reddy Bhimireddy, John Burns, Leo Anthony Celi, Li-Ching Chen, Ramon Correa, Natalie Dullerud, Marzyeh Ghassemi, Shih-Cheng Huang, Po-Chih Kuo, Matthew P Lungren, Tyle J Palmer, Brandon J Price, Saptarshi Purkayastha, Ayis T Pyrros, Lauren Hakden-Raynen Chima Okechukwu, Laleh Seyyed-Kalantari, Hari Trivedi, Ryan Wang, Zachary Zaiman, and Haoran Zhang. Ai recognition of patient race in medical imaging: a modelling study. The Lancet Digital Health, 2022. ISSN 2589- 7500. doi: /doi.or///o.org/////// /t..or/////...... 10.1016 92589-7500
URL httpp::/www. scienceddrect. wwwp://dwwweeettccccceeeeeeoeeee cww/sciincc///////cc/ccccccccccc
g
[41] A.A. Minai and R.D. Williams. Perturbation response in feed-forward neural networks. In (Proceedings 1992] IJCNN International Joint Conference on Neural Networks, volume 3, pages 857-862 vol.3, 1992. doi: 10. 11L10//11CN.199222222
3
[42] Wenjie Ruan, Min Wu, Youcheng Sun, Xiaowei Huang, Daniel Kroening, and Marta Kwiatkowska. Global robustuess evaluation of deep neural networks with provable guarantees for the hamming distance. In Proceedings of the Snventy-Eighll International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5952-5952. International Joint Conferences on Artificial Intelligence Organization, 7 2019. doi: 10 wt..1666//ciii2001////
URL https: //doii org/10 1et11..2993/iaaaa... /098////caa.2211/882 2019/824
[43] Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and
perturbations. Proceedings of the International Conference on Learning Representations, 2019. g
22
[44] Fuxun Yu, Zhuwei Qin, Chenchen Liu, Liang Zhao, Yanzhi Wang, and Xiang Chen. Interpreting and evaluating neural network robustness. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJ page 4199-4205. AAAI Press, 2019. ISBN 9780999241141.
[45] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[46] Nimit Sohoni, Jared Dunmmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. Advanees in Neurul Information Processing Systems, 33:19339-1190 19002,9992222 2020.
[47] Karan Goel, Albert Gu, Yixuan Li, and Christopher Re. Model patching: Closing the subgroup performance gap with data augmentation. International Conference on Learning Representations, 2021. URL, https
//openreview. wttp://www..eeeeoeeeredeooeeeeee
[48] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance regularization for worst-case generalization, arXiv preprint arXi::1911.08711, 2019.
[19] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovee, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finu, and Percy Liang, Wilds: A beuchmark of in-theewwild distribution shifts. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5637-5664. PMLR, 18 24 Jul 2021. URL html
https://prooeedingg.. mlr. wtw..rpreess//////eeeeee
23

---

Published in Transactions on Machine Learning Research (05/2023)
[50] Francesco Croce, Maksym Audriushchenko, Vikush Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateck Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustuess benchmark. In Thirty-filth Conference on Neural Information Processing Systems Dalasels and Benchmarks Track (Round 2), 2021.
[51] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances Neural Information Processing Systems, 33:11539-11551, 2020.
[22] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances neural information processing systems, 32, 2019.
[53] Saul Calderon Ramirez, Luis Oala, Jordina Torrentes Bartte,,,ar Shengxiang Yang, David Elizondo, Armaghan Moemeni, Simon Colreavy- Wojciech Samek, Miguel Datasel similarity to assess semiisupervised learning under distribution mismatch between the labelled and unlabelled datasets. IEEE Transactions on Artificial Intelligence, 2022.
and Ezequiel Lopez- Rubio.
[54] Frederick M Howard, James Dolezal, Sara Kochamny, Jefree Schulte, Heather Chen, Lara Heij, Dezheng Huo, Rita Nanda, Olufummilayo I Olopade, Jakob N Kather, et al. The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nature commanications, 12(1):1-13, 2021. 3
[55] Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-We Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. Advances in Neural Information Processing Systems, 33:1129 1141, 2020. 3 19
[56] Linyi Li, Xiangyu Qi, Tao Xie, and Bo Li. Sok: Certified robustness for deep neural networks. arXiv preprint ar Xiv:20000.00 14131, 2020.
[57] Runtian Zhai, Chen Dan, Di He, Huan Zhang, Boqing Gong, Pradeep Ravikumar, Cho-Jui Hsieh, and Liwei Wang. Macer: Attack-free and scalable robust training via maximizing certified radius. In International Conference on Learning Representations, 2019.
[58] Jongheon Jeong and Jinwoo Shin. Cousistency regularization for certified robustness of smoothed classifiers. Advances in Newral Information Processing Systems,
2020.
(59] Hadi Salman, Greg Yang, Huan Zhang, Cho-Jui Hsieh, and Pengchuan Zhang. A convex relaxation barrier to tight robustness verification of neural networks. Advances én Neural Information Processing Systems, 32, 2019.
[60] Sven Gowal, Krishnamurthy Dj Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. Scalable verified training for provably robust image classification. In Proceedings of the IEEE/CVE International Conference Computer Vision, pages 1842 4851, 2019.
[61] Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers, Advonces in Neural Information Processing Systems, 32, 2019.
[22] Sumanth Dathathri, Krishnamurthy Dvijothmm, Alexey Kurakin, Aditi Raghunathan, Jonathan Uesato, Rudy R Bunel, Shreya Shankar, Jacob Steinhardt, Ian Goodfellow, Percy S Liang, et al. Enabling certification of verification-agnostii networks via memory-efficiett semidefinite programming. Advances in Neural Information Processing Systems, 33:5318 5331, 2020.
[63] Jeremy Cohen, Flan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pages 1310-1320. PMLR, 2019.
[64] Zhuolin Yang, Linyi Li, Xiaojun Xu, Bhavya Kailkhura, and Bo Li. On the certified robustness for ensemble models and beyond, 2021. URL httpp:://pperrevier http://wwwww..eeeo.o////////ta/a
[65] Julian Bitterwolf, Alexander Meinke, and Matthias Hein. Certifiably adversarially robust detection of out-of- distribution data. Advances in Neural Information Processing Systems, 33:16085 16095, 2020.
[66] Shafi Goldwasser, Guy N Rothblum, Jonathan Shafer, and Amir Yehudayoff. Interactive proofs for verifying machine learning. In 12th Innovations in Theoretical Computer Science Conference (ITCS 2021). Schloss Dugstuuh-LLiibuizzzZeennuuu für Informatik, 2021.
667] Shafi Goldwasser, Adam Tauman Kalai, Yael Kalai, and Omar Montasser. Beyond perlurbations: Learning guarantees with arbitrary adversarial test examples. Advances in Nearal Information Processing Systems, 33: 2020.
24

---

Published in Transactions on Machine Learning Research (05/2023)
[68] Adarsh Subbaswarny, Roy Adamss, and Suchi Saria. Evaluating model robustness and stability datasel shift. In International Conference on Artificial Intelligence and Statislics, pages 2611-2619, PMLR, 2021. G
[69] Mayee Chen, Karan Goel, Nimit S Sohoui, Fait Pous, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In International Conference on Machine Learning, pages 1617-1629 PMLR, 2021.
[70] Patrick Cousot. Abstract interpretation. ACM Computing Surveys (CSUR), 28(2):324-33) 328, 1996.
[71] Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. A12: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEFE Symposium on Security and Privacy (SP), pages 3-18. IEEE, 2018. 19
[72] David A Nix and Andreas S Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 feee international conference on neural networks (ICNN'94) volume 1, pages 55-60. 18
IEEE, 1994.
[33] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9
(3); 2008.
[74] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050-1058 1059, New York, New York, USA, 20-22 Jun 2016. PMLR. URL httttt://ppoocceeeeeeee
.mlr htress//8/// //////all6..... html
[75] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in nearal information processing systems, 30, 2017.
[76] Jochen Gast and Stefan Roth. Lightweight prohabilistic deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3369-3377 2018.
[77] Hartmut Maennel. Uncertainty estimates and out-of-distribution detection with sine networks. 2019.
[78] Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In International conference on machine learning, pages 9690-9700. PMLR, 2020.
[79] Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33:7498 7512, 2020.
[80] Luis Oala, Cosmas HeiB, Jan Macdonald, Maximilian März, Gitta Kutyniok, and Wojeiech Samek. Detecting failure modes in image reconstructions with interval neural network uncertainty. International Journal of Computer Assisted Radiology and Surgery, 16(12):2089-2000
2021.
[81] Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael Dusenberry, Sebastian Farquhar, Angelos Filos, Marton Havasi, Rodolphe Jenatton, Ghassen Jerfel, Jeremiah Liu, Zelda Mariet, Jeremy Nixon, Shreyas Padhy, Jie Ren, Tim Rudner, Yeming Wen, Florian Wenzel, Kevin Murphy, D. Sculley, Balaji Lakshminarayauan, Jasper Snoek, Yarin Gal, and Dustin Tran. Uncertainty Baselines: Benchmarks for uncertainty & robustness in deep learning. arviv preprint ar Xrx:: 10m:2106.04015 2021.
82) Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint or 2rXi:21077075111 2021. g T
[83]
Dipankar Dasgupta. Artificial immume systems and their applications. Springer Science Business Media, 2012.
3
[84]
Skyler Speakman, Sriram Somanchi, Edward McFowland III, and Daniel B Neill. Penalized fast subset scanning. Journal of Computational and Graphical Statistics, 25(2):382-404, 2016.
[85] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
[86] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprind 18100 0.09136, 2018
[87] Xuefeng Du Zhawning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. In Intervational Conference on Learning Representations, 2021.
3 18
[88] AGB Oliver, AGB Odena, CGB Raffel, EGB Cubuk, and IJGB Goodfellow. Realistic evaluation of semi- supervised learning algortihus. In International conference on Learning Representations, pages 1-15, 2018. 3
18
25

---

Published in Transactions on Machine Learning Research (05/2023)
[89]
Qiahe Xie, Zihang Dai, Eduard Hovy, Thang Luoug, and Quoc Le. Unsupervised data augmentation for consistency training, Advances in Neural Information Processing Systems, 33:6256 6268,,2668882008 2020.
[90]
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch:
A holistic approach to semi-supervised learning- Advonces in Neural Information Processing Systems, 32, 2019.
3
18
[91]
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good
views for contrastive learning? Aduances in Neural Information Processing Systems, 33:6827-6888 6839, 2020. T8
[22]
Wean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 2222814, 2020.
3 18
[93] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning
(still) requires rethinking generalization. Communications of the ACM, 61(3):107 115, 2021.
[94]
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389 54100. PMLR, 2019.
[95] Mahdi Haghifam, Gintare Karolina Dziugaite, Shay Moran, and Dan Roy. Towards a unified theooeiiin-thhereticc framework for generalization. Advonces in Neural Information Processing Systems, 34, 2021.
[96]
Krikamol Muandet. Impossibility of collective intelligence, 2022. URL hatps:://arxti wwt. 2206. 02786
17
[97] Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojeiech Samek. Robust and communication- efficient federated learning from non-iid data. IEEE transactions on neural networks and learning systemss 31 (9):3410 3413, 2019.
3 19
[98] Felix Sattler, Klaus- Robert Muller, Thomas Wiegand, and Wojeiech Samek. On the byzantine robustness of clustered federated learning. In ICASSP 2020-2020 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
IEEE, 2020.
(99] Hao Wang, Zakhary Kaplan, Di Niu, and Baochun Li. Optimizing federated learning on non-iid data with reinforcenent learning. In IEEE INFOCOM 2020 IEEE Conference on Computer Communicatiins,, pages 1707, 2020. doi: 10. 10.1109/INFOOO ////OOOOOOOO1// SAA00CCOAAA0442220999555555
3 19
[100] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXir preprint
vlysics/0004057 2000. g
[101] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with demoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103, 2008
[102] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXin: 1101880033555 2018.
[103] Georg Krempl, Vera Hofer, Geoffrey Webb, and Eyke Hüllermeier. Beyond Adaptation: Understanding Distributional Changes (Dagstuhl Seminar 20372). Dagstuhl Reports, 10(4):1-36, 2021. ISSN 2192-5283. doi:
De.30//DDRee.1100 10.4.1. URL https:///ropp: http://drooss.ddeuuuulcccoc deepeaaadaatttaleeppppppeee..eee httee//ppne/lllttttt////////////
3
[104] Peter J Huber. Robust statistics. In International encyclopedia of statistical science, pages 1248-1251. Springer, 2011.
[105] AAMI TIR57. Principles for medical device security risk management. Arlington, VA: Association for the
A Adbancement of Medical Instrumentation, 2016. 3
14 19
[106] IMDRF SaMD Working Group et al. Software as a medical device (samd): Application of quality management system, 2018.
[107] Luis Oala, Jana Fehr, Luca Gilli, Pradeep Balachandran, Alixandro Werneck Leite, Saul Ramirez Danny Xie Li, Gabriel Nobis, Erick Alejandro Muñoz Alvarado, Giovanna Jaramillo-Cutierre, et al. MMh auditing: From paper to practice. In Machine learning for health, pages 280 317. PMLR, 2020. 3
[108] Emma Beede, Elizabeth Baylor, Fred Hersch, Anna lurchenko, Lauren Wilcox Paisan Ruamviboonsuk, and Laura M. Vardoulakis. A humaa--entered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In Proceedings of the 2020 CHI Conference on Humon Factors in Computing Systems, page 1-1 New York, NY, USA, 2020. Association for Computing Machinery. ISBN 4781450367000.
URL htttps:///o: www.oiii.rg//.... 111111141111881 33767188
3
26

---

Published in Transactions on Machine Learning Research (05/2023)
[109]
Susan M. Swelter. Artificial intelligence may improve melanoma delection.
Dermalol-
ogy Tines, 41(9):36, 2020.
URL https://cdn.... http:://dn...anttt....... http/////////aaa/////////aoo////
pdf
[110] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-leeel classification of skin cancer with deep neural networks. nature, 342(7639):115-11)
2017. 3
[111]
Maitiniyazi Maimaitijiang, Vasit Sagan, Paheding Sidike, Sean Hartling, Flavio Esposito, and Felix B. Fritschi. Soybean yield prediction from using multimodal data fusion and deep learning. Remote Sensing of Environment, 237:111599, 2020. ISSN 0034-4257. doi:
URL
44
[112] Phillip Chlap, Min Hang, Nym Vandenberg, Jason Dowling, Lois Holloway, and Annette Haworth. A review of medical image data augmentation techniques for deep learning applications. Journal of Medical Imaging and Radiation Oncology, 65, 06 2021. doi: (0.1111/1754-9486.1
X
[113] Christian Matek and Carsten Marr. Robustness evaluation of a convolutional neural network for the classification of single cells in acute myeloid leukemia. In JCLR 2021, RobustML workshop, 2020.
114] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In Advences in Neural Information Processing Systems (NewrIPS), 2020. URL
12
[115] GJJ Verhoeven. It's all about the formattuuleashing the power of raw aerial photography. International Journal of Remote Sensing, 31(8):2009-2044 2010. 4
[116] Ronnachai Jaroensri, Camille Biscarrat, Miika Aittala, and Frédo Durand. Generating training data for denoising
real rgb images via camera pipeline simulation. orXiv, 1904.08825, 2019. 4 12
[117] B Albertina, M Watson, C Holback, R Jarosz, S Kirk, Y Tee, and J Temmerman. Radiology data from the cancer genome atlas lung adenocarcinoma [tega-luad] collection. The Cancer Imaging Archive, 2016. F
[118] C Matek, Schwarz, C Marr, and K Spiekermann. A single-cell morphological dataset of leukocytes from aml patients and non-malignant controls (aml-cytoomophhologyy. Imm). The Cancer Imaging Archive (TCIA), 2019.
119] Weixin Liang and James Zou, Metashift: dataset of datasets for evaluating contextual distribution shifts and training conflicts. In International Conference Learning Represenlations, 2022. URL https:://ppenrevier
net/forummiid-MMeezzaaaaar
4
[200]
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "everyone wants to do the model work, not the data work": Data cascades in high-stakes ai. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 450380966. URL (tTLhhttaa://// /ttp::::///ooo.r/111.... 1145/3411166 3445518
6
[21]
Bryce E Bayer. Color imaging array, July 1976. US Patent 3,971,065,
[22]
Andy Rowlands. Physics of digital photography. TOP Publishing, 2017. 7
[123]
Shinsuke Tani, Yasuhiro Fukunaga, Saori Shimizu, Munenori Fukunishi, Kensuke Ishii, and Kasei Tamiya, Color Standardization Method and System for Whole Slide Imaging Based Spectral Sensing. Analytical
Cellular Pathology, 3572):107-115, 2012. ISSN 2210-7177, 2210-7185. doi: 10.1155/2012/155 114435 URL
http://wwz hwwp://wwwiwwwiioommmm wwwpnddwwi.co//ouuuuuu/oaooooooe http://oonnaas/cc//11002.2/ 15447//
[124] Daniel L. Bongiorno, Mitch Bryson, Donald G. Dansereau, and Stefan B. Williams. Spectral characterization of COTS RGB cameras using a linear variable edge filter. page 86600N, Burlingame, California, USA, January 2013. doi: (0.1117/12.2001466 URL http::/pproceedingg http:p/ppoooeeinneeeeeeeeggaaaaa htt.iiiiiallbbbrr...or/rereeeeee aspx?doi=
10.1117/12.2001
[125]
HAMAMATSU. ORCA-Flash4.0 V3 Digital CMOS camera C13410-20CU Technical note. HAMA-
MATSU. URL attps://wwx: hww.wata..aacmmmmmmm...o.aa..aaa cww/ http://awwcaacceeeeeeee//otteeee
html#
eemeett - 1d-
19
[126]
PerkinElmer. TotaIChrom Workstation User's Guide Volume PerkinElmer. URL http:::///u....riiinn....
com/ Cww.eessurree//naaaaaaeae 744577M wtw.aa..crroooorrttttttooooooooa .pff
ZEISS. [127]
Eaporting
Images and
Movics in ZEN Bluc.
ZEISS.
URL
https://
mmorting-imagenn www.gee--and-moiii-iiini exportigggiinggerrracrreeeeeeeee htwp//dwwolld///// wwttmnloddddd//eeeeeeee ottp:///ooooottaaaacccccootooooo wewpreeedccaaiiincctetteeee center/
www. zwiss. com/ https////cccccttteeeea///eeteoee
27

---

Published in Transactions on Machine Learning Research (05/2023)
[128] De Jong Yeoug, Gustavo Velasco-Heruandez, John Barry, and Joseph Walsh. Sensor and sensor fusion technology in autonomous vehicles: A review. Sensors, 21(6), 2021. ISSN 1424-8220. doi: 10.3390/s220621140. URL
nttps://www.ndpp http://www.ddi..oo///22/////
[129]
Audrej Karpathy. Tesla ai day 2021. URL
[130] Gert Rudolph and Uwe Voelzke. Three semsor types drive aulonomous vehicles, 2017. URL https://wwx
fierceelectronics. wwwp:///oopooetttttteeaeeaaeeeea
19
[131]
Nai-Sheng Syu, Yu-Sheng Chen, and Yung- Yu Chuang. Learning deep convolutional networks for demosaicing.
arXiv, 1802.03769, 2018. G
[32]]
S. Ratnasingam. Deep camera: A fully convolutional neural network for image signal processing. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3868 3878, Los Alamitos, CA, USA, oct 2019. IEEE Computer Sociely. doi: http:////C/.221...00200 URL
https:///di. jeeecconputerssciiety......... wrtpt/..rr//... 1109/ICCVV.
6
[133] Ethan Tseng, Ali Mosieh, Fahim Mannan, Karl St-Arnaud, Avinash Sharma, Yifan Peng, Alexander Braun, Derek Nowronzezahrai, Jean-Frangois Lalonde, and Felix Heide. Differentiable compound optics and processing pipeline optimization for end-to-end camera design. ACM Trans. Graph., 40(2), June 2021. ISSN 0730-0301.
doi: 10.1145/3440 33447914 URL httttt::/toi..orreeeee wttp 1145/34469991
[134] Jonathan Ragan-Kelley, Counelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing
pipelines. Acm Sigplan Notices, 48(6):519 530, 2013. 6
[135] F. Riba, D. Mishkin, D. Ponsa, F. Rublee, and G. Bradski. Kornia: an open source differentiable computer vision library for pytorch. In Winter Conference on Applications of Computer Vision, 2020. URL https:
wtg/pdf////// //910. 02w.0.299p... pdf] G
[136] Felix Schill. pyraw. httppe///gthhh.ccccc/aaaaa//paap 2015. 6
coomffcchill/pyrar
[137] Andreas K Maier, Christopher Syben, Bernhard Stimpel, Tobias Würff, Mathis Hoffmann, Frank Schebesch, Weilin Fu, Leonid Mill, Lasse Kling, and Silke Christiansen. Learning with known operators reduces maximum
error bounds. Nature machine intelligence, 188):373-38 -0333300, 2019. 6
[138]
Andreas Maier, Harald Köstler, Marco Heisig, Patrick Krauss, and Seung Hee Yang. Known operator learning and hybrid machine learning in medical imaging review of the past, the present, and the future. Progress in Biomedical Engineering. 2022. C
[139] Tatiana A Bubba, Gitta Kutyniok, Matti Lassas, Maximilian März, Wojeiech Samek, Samuli Siltanen, and Vignesh Srinivasan. Learning the invisible: hybrid deep learning-shearlet framework for limited angle computed tomography. Inverse Problems, (566)0064 2019. G
[140] Bo Zhu, Jeremiah Z7 Liu, Stephen F Cauley, Bruce R Rosen, and Matthew S Rosen. Image reconstruction by transforr manifold learning. Nature, 565(7697):487-492, 2018 6
[141] Samuel Dodge and Lina Karam. study and comparison of human and deep learning recognition performance under visual distortions. In 2017 26th International Conference Computer Communiccttin and Networks (ICCCN), pages 1 7, 2017. doi: 10.1109/ICCCN, 110.0//CCC...211...3888/// 2037.8038 G
[122] Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly small image transformations? Journal of Machine Learning Research, 20:1-25, 2019. 6
[143] Joseph Paul Cohen, Margaux Luck, and Sina Honari. Distribution matching losses can hallucinate features in medical image translation. In Medical Image Computing and Computer Assisted Intervention -ICCAI 2018, pages 529-536. Springer International Publishing, 09 2018. ISBN 078-3-030-00928-1. doi: 10.1007/
978-3-030-00928-11 60. 0
[144] Florian Schiffers, Zekuan Yu, Steve Arguin, Andreas Maier, and Qiushi Ren. Synthetic fundus fluorescein angiography using deep neural networks. Andreas Maier, Thomas M. Deserno, Heinz Handels, Klaus Hermann Maier-Hein, Christoph Palm, and Thomas Tolxdorff, editors, Bildverarbeitung für die Medisin 2018, pages 234-238, Berlin, Heidelberg, 2018. Springer Berlin Heidelberg. ISBN 778-3-662-56633-7. 6
145] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition terra incognita. In Proceedings of the European Conference on Compuler Vision (ECCV), pages 456-475333 2018. B
[146] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In Proceedings
28

---

Published in Transactions on Machine Learning Research (05/2023)
of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Muchine Learning Research, pages 5637-560 5664. PMLR, 2021. URL |ateps://proceediingg mlr wttppp/e1e///eeeaeeeaa html
[147]
Buu Phan, Fahim Mannan, and Felix Heide. Adversarial imaging pipelines. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16051-16061, 2021.
7 15
[148]
Matteo Ronchetti. Torchradon: Fast differentiable routines for computed tomography. arXis, 2009.14788, 2020.
f
[149]
Christopher Syben, Markus Michen, Bernhard Stimpel, Stephan Seitz, Stefan Ploner, and Andreas K Maier.
Pyro-mm: Python reconstruction operators in neural networks. Medical physics, 16(11):5110 5115, 2019.
[150]
Andreas Maier, Harald Köstler, Marco Heisig, Patrick Krauss, and Seung Hee Yang. Known operator learning and hybrid machine learning in medical imaging a review of the past, the present, and the future. arXiv, 2108. 04543, 2021.
[151]
Rang Nguyen, Dilip Prasad, and Michael Brown. Raw-to-raw: Mapping between image sensor color responses. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3398-3405, 2014.
[152] Library of Congress. Camera Raw Formats (Group Description).
wwt.::/ooo.ooo.oooooooooo URL https://www.lo.e. https://ww...ooccco///rrrrrraata
formatas/fdd/ffd0002411 shhmm
December 2016.
formatas/Edd/FFdd000244 shtml
Accessed: 2020-11-03.
7
[153]
Sumona Biswas and Shovan Barma. A large-scale optical microscopy image dataset of potato tuber for deep learning based plant cell assessment. Seientific Data, 7(1):1 11, 2020. URL https:://doo.org/00/
-41597-020-00706-9
[154]
Ryan Conrad and Kedar Narayan. Cem500k, large-scale heterogencous unlabeled cellular electron microscopy image dataset for deep learning, eLife, 10:c65894, apr 2021. ISSN 2050-084X. doi: (11/54//eiff.55555 URL
nttps://dbi.org/ 10. 7554/eLife. 65894
[155]
Guilherme Aresta Teresa Araújo, Scotty Kwok, Sai Saketh Chennamsetty, Mohammed Safwan, Varghese Alex, Bahram Marami, Marcel Prastawa, Monica Chan, Michael Donovan, Gerardo Fernandez, Jack Zeinch, Matthias Kohl, Christoph Walz, Florian Ludwig, Stefan Braunewell, Maximilian Baust, Quoc Dang Vu, Minh Nguyen Nhat To, Eal Kim, Jin Tae Kwak, Sameh Galal, Veronica Sanchez-Freire, Nadia Brancati, Maria Frueci, Daniel Riccio, Yaqi Wang, Lingling Sun, Kaiqiang Ma, Jiannan Fang, Ismael Kone, Lahsen Boulmane, Aurélio Campilho, Catarina Eloy, António Polónia, and Paulo Aguiar. Bach: Grand challenge on breast cancer histology images. Medicall Image Analysis, 56:122-139, 2019. ISSN 1361-8415. doi: https:/ http:///ooooooooii// 10.11.0.00000 htt.:..media.2019.000
URL http:::://ww...iiiccciie.....e.. www.sscciiinncccc..ccmr com/ hwwp://cce.crrett..coocccceeeeee httssscciicccarreeee /tt.///c/oaaaat.c.//////////.11/
[156] Le Hou, Rajarsi Gupla, John S Van Arnam, Yuwei Zhang, Kaustubh Sivalenka, Dimitris Samaras, Tahsin M Kurc, and Joel H Salta. Dataset of segmented nuclei in hematoxylin and eosin stained histopathology images of cancer types. Scientific data, 1(1):1-12, 2020.
[157] Hamidreza Bollasaai, Elham Amjadi, Maryam Tabatabacian, and Somayyeh Jafarali Jassbi. A lustooaathologicall image dalaset for grading breast invasive ductal carcinomas. Informatics Medicine Unlocked, 19:100341, 2020. ISSN 2352-9148. doi:
wtt.://///...../...o.oo 200.22000.11011.. URL. http:://www.cceencedrerrr sciencedirect
[158] TU Graz. ICG DroneDatasel. httpp://www. www.://wgraz.cco htw.gaaa.tt/nnder..ppiiii----- 2019. URL https://www //vuw
aigrra..t//nde..pp222422233 Accessed: 2021-05-06.
[159] DroneDeploy.
Segmentation Dataset. htttpe://ggtaabb.cmm/drreeeeeeee com/
2019. URL http:s///gshhib...c/////dddppppp http:///w......ooo///oooaaaaaata
Accessed: 2021-09-19.
[160] Ye Lyu, George Vosselman, Gui-Song Xia, Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic
segmentation dataset for uav imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 155:108-199
2020. ISSN 0924-2716. doi: https: /toi.org///////////////o 10. http://.spprppp............. URL ltttp::// httpp::/uww..ciinnedireee scienceireetc hotp://tsaaae///rrraaa/////////2
[161] Alina Marcu, Dragos Costea, Vlad Licaret, and Marius Leordeanu. Towards automatic annotation for semantic segmentation in drone videos. Xrv, 1910.10026, 2019.
[162] Alireza Shamsoshoara, Fatemeh Afghah, Abolfazl Razi, Liming Zheng, Peter Z. Fulé, and Erik Blasch. Aerial imagery pile burn detection using deep learning: The flame dataset. Computer Networks, 193: 108001, 2021. ISSN 1389-1286. doi: https://doi..or 10.11100.1.11.1111111111111 (016//.commet.200. 2080000000.. URL httppp////www.scceeeeeree.e.... com/
[163] P. Zhu, L. Wen, D. Du, X. Bian, H. Fan, Q. Hu, and H. Ling. Delection and tracking meet drones challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence, (01):1-1, October 2021. ISSN 1939-3539. doi: 10. 0.1109/TPAMM.2001 3119563.
29

---

Published in Transactions on Machine Learning Research (05/2023)
[164] Duc-Tien Dang-Nguyen, Cecilia Pusquini, Valentina Conotter, and Giulia Boato. Raise: A raw images datasel for digital image forensies. In Proceedings of the 6lh ACM Mullimedia Systems Conference, MMSys '15, page 219-224, 224, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450333511. doi: 10.1145/271316. 2713194. URL btttpa://doi.oo 1145/77711/ 27731994
7
[655] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. 2018 TEEE/CVP Conference on Computer Vision and Pattern Recognition, pages 3291-3300, 2018.
[166]
Abdelrahman Abdelhamed, Stephen Lin, and Michael S. Brown. A high-quality denoising dataset for smartphone cameras. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[167] Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Mare Levoy. Burst photography for high dynamic range and low-light imaging on mobile cameras.
ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 35(6), 2016. 7
1668] Robert William Gainer Hunt and Michael R Pointer. Measuring colour. John Wiley & Sons, 2011.
[169] Xin Li, Bahadir Gunturk, and Lei Zhang, Image demosaicing: A systematic survey. In Visual Commuunicationss and Image Processing 2008, volume 6822, page 68221J. International Suciely for Oplics and Photonics, 2008. 7
[170]
Bhawna Goyal, Ayush Dogra, Sunil Agrawal, BS Soli, and Apoorav Sharma. Image denoising review: From classi- cal to state-of-the-art approaches. Information Pusion, 55:220 244, 2020. ISSN 1566 2535. doi: https:/ /ttp://ddooorrgggg 10. 1016/j (016//.inffus.221 inffus.2011 201...00100000 9.009.003 URI. http::///www..cenneeeiie /ttp://///cccaaaaaaaaaaaaaaa/aaa
[11] Chunwei Tian, Lunke Fei, Wenxian Zheng, Yong Xu, Wangmeng Zuo, and Chia- Wen Lin. Deep learning on image denoising: An overview. Neural Networks, 131:251-275, 2020. ISSN 0893 6080. doi: https://doi.orr 10. 1016/j. neumet.2020.07 .055. URL http::::/wwwsscieeeedeeee
X
[172] Mathieu Dejean-Servières, Karol Desnos, Kamel Abdelouahab, Wassim Hamidouche, Luce Morin, and Maxime Pelcat. Study of the impact of standard image compression techniques on performance of image classification with a convolutional neural network. Research Report hal-01725126, INSA Rennes; Univ Rennes; IETR; Institut Pascal, 2017.
[173] Yong- Yeon Jo, Young Sang Choi, Hyun Woo Park, Jae Hyeok Lee, Hyojung Jung, Hyo-Eun Kim, Kyounglan Ko, Chan Wha Lee, Hyo Soung Cha, and Yul Hwangbo. Impact of image compression on deep learning- based mammogram classification. Scientific Reports, 11(1):1-9, 2021.
174) Farhad Ghazvinian Zanjani, Svitlana Zinger, Bastian Piepers, Saeed Mahmoudpour, Peter Schelkens, and Peter H. N. de With. Impact of JPEG 2000 compression on deep convolutional neural networks for metastatic cancer detection in histopathological images. Journal of Medical Imaging, 6(2):1 9, 2019. doi: 10.1117/L.JM...22222773. URL htttps://doii org/10. JM1 6. 227501
[175] Matt Poyser, Amir Mtaour-Abeet Abarghouei, and Toby P Breckon. On the impact of lossy image and video compression on the performance of deep convolutional neural network architectures. In 2020 25th International Conference on Pottern Recognition (ICPR), pages 2830-2837. IEEE, 2021.
[176] Enrico Pomarico, Cédric Schmidt, Florian Chays, David Nguyen, Arielle Planchette, Audrey Tissot, Adrien Roux, Laura Batti, Christoph Clausen, Theo Lasser, el al. Statistical distortion of supervised learning predictions in optical microscopy induced by image compression. Scientific reports, 12(():1-10, 2022,
[177] Timmit Gebru, Jamie H. Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé, and Kate Crawford. Datasheets for datasets. arNio, 1803.09014 2018.
46
[178] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron. Unprocessing images for learned raw denoising. IEEE Conference on Computer Vision ond Pattern Recognition (CVPR), 2019. 9
[179] Seonghyeon Nam, Abhijith Punnappurath, Marcus A. Brubaker, and Michael S. Brown. Learning srgb-to-raw-rgh de-rendering with content-aware metadata. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17704 17713, June 2022.
g 0
[180] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi:
10.1109/CVPR.200 200.1...9.... I1 37
[181] Olaf Ronneberger, Philipp Fischer, and Thomas Brox U-net: Convolutional networks for biomedical image seguentation. In International Conference on Medical Image Computing and Compuler- Assisteed Intervention,
pages 234 241. Springer, 2015.
11 37
30

---

Published in Transactions on Machine Learning Research (05/2023)
[182] USDOT NSTC. Eusuring american leadership in automated vehicle technologies: Automated vehicles 4.0. Las
Vegas. Recuperado el, 25:2020- 02, 2020. 15
[183] Martin Genzel, Jan Macdonald, and Maximilian Marz. Solving inverse problems with deep neural networks
robustness included. IEEE Transactions Pattern Analysis and Machine Intelligence, pages 1-1, 2022. doi: N..109/TPAAM.2022..22
16
[184]
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):151 1155, 2010. 19
[185] Logan G Wright, Tatsuhiro Onodera, Martin M Stein, Tianyu Wang, Darren T Schachter, Zoey Hu, and Peter L McMahon. Deep physical neural networks trained with backpropagation. Nature, 601 (78944)::5449 555, 2022.
19
[186]
Diederik P. Kingma and Jimmyy Ba. Adam: A method for stochastic optimization. CoftR, 1412.6980, 2015.
37
[187]
Olga Russakovsk,, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vision, 115(3):211 2522, dec 2015. ISSN 0920-5691. doi: 10.1007/ $11263-015-0816-y. URL https::/doi.
1007 10111000000001111222---------110
37
[188] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. orXiv preprint arXin: I 1903. 12261, 2019. 43
[899] B. Bain. Diagnosis from the blood smear. The New England journal of medicine, 353 5:498 507, 2005. 44
[190] Ruggero Donida Labati, Vincenzo Piuri, and Fabio Scotti. All-idb: The acute lymphoblastic leukemia image database for image processing. In 2011 18th IEEE International Conference Image Processing, pages 2045 2048, 2011. doi:
14
191] Vinay Ayyappan, Alex Chang, Chi Zhang, Santosh Kumar Paidi, Rosalie Bordett, Tiffany Liang, Ishan Barman, and Rishikesh Pandey. Tdentification and staging of b-cell acute lymphoblastic leukemia using quantitative 2020, doi: 10.1021 Cetesssnssors.0011881
phase imaging and machine learning. ACS Sensors, PMID: 33092347. 44
URL httts:://oo org/10. 1021/acssensors 0c01811
[192] David Tellez, Geert Litjens, Péter Bándi, Wouter Bulten, John-Melle Bokhorst, Francesco Ciompi, and Jeroen van der Laak. Quantifying the effects data augmentation and stain color normalization in convolutional neural networks for computational pathology. Medical Image Analysis, 58:101544, 2019. ISSN 1361-8415. doi:
htt.:/oogg//11100/..meeiei 201...20119..1 URL
$1361841519300799 +
[193] S Longanbach, MK Miers, EM Keohane, LJ Smith, and JM Walenga. Rodak's hematology: Clinical principles and applications. 2016. 44
[194 Marek Kulbacki, Jakub Segen, Wojeiech Knieć, Ryszard Klempous, Konrad Kluwak, Jan Nikodem, Julita Kulbacka, and Audrea Serester. Survey of drones for agriculture automation from planting to harvest. In 2018 IEEE 22nd International Conference on Intelligent Engineering Systems (INES), pages 000353 000358, IEEE, 2018.
44
[195] Xiyue Jia, Yining Cao, David O'Connor, Jin Zhu, Daniel CW Tsang, Bin Zou, and Deyi Hou. Mapping soil pollution by using drone image recognition and machine learning an arsenic-contaminated agricultural field. Environmental Pollution, 270:116281, 2021. 44
31

---

Published in Transactions on Machine Learning Research (05/2023)
A Appendices
A.1 Data models details
A.1.1 Static data model stat
If not stated otherwise, writing the equation Ve,h, ae,h, be, defines Ve for all 1s sshs and 1 S h W.
Black level correction (BL) removes thermal noise and readout noise generated from the camera sensor. The transformation is given by
T BL [0, [0, H. DRAW UBL,
(10)
with
(UBL)22+ = 1,20-1 11
(H,T.)25,204+ = 222h, bl2
2 -22h-1,2w
blg
UBL)2h,22r 22h,
bls.
By design of bl E R', black level correction ensures that UBL is again an element of [0, [J.
Demosaicing (DM) is applied to reconstruct the full RGB color image through interpolation. We use one Men out of the three demosaicing algorithms BayerBilinear ( (M)) Menon2007 ( DM and Malvar2004 Mal
from the Python package selor-demosaccing and denote this transformation by the map
PpM [0. 1 [0, HJS.H,WW UDM
(11)
White balance (WB) is applied obtain a neutrally illuminated image. The transformation given by Pwn [0,1] + [0, JJ3.H.W UWB.
(12)
where wh e [0, adjusts the intensities by
(UWB)o, who (VDM)c,,,.
(13)
Color correction (CC) balances the saturation of the image by considering color dependencies. M 333 be the color matrix. The transformation is defined by
Let
Pcc [0, 10, + H.H... 0 VCC,
(14)
where
(UCc)
(UWB)
VCc =
M
(1WB)2,,,00
(15)
(vCc) h,
(DWB)
The entries of the resulting voc are no longer restricted to [0.1].
Sharpening (SH) reduces the blurriness of an image. We use the two mothods sharpening filter ($SH) and unsharp masking & (H that are applied after transformation of the view toc to the YUV-color space.
To convert the view to the YUV-color space we use the Ekimage.colme color function rgb2yuv (œyov) The sharpening filter
SF: SF:R3.H. R3H.W
(16)
is defined by channel-wise convolution
0 -1 0
(SF(v)) = ((v, k) k)nw) with =
-1 -1
5
(17)
0
32

---

Published in Transactions on Machine Learning Research (05/2023)
of the view
x= yvv (vcc).
(18)
For unsharp masking we use the oki.filters function unsharp_ mask modeled by UM. To formally define the sharpening we write
PHH R.H.W.WW W...H.W u USH
(19)
where
USH algoo @yuv with algo ()..
(20)
Denoising (DN) reduces the noise in an image that is (partly) introduced by SH and transforms the YUV-color space view back to the RGB-coloo space. For the latter transformation, the skimage. color used. We apply one out of the two methods Gaussian denoising ($DN) and Median denoising (C.)) ). For Gaussian denoising, we apply a Gaussian filter (GF) with standard deviation of 0.5 from the scipy. .eiiagee package. For median denoising we apply median filter (MF) of size 3
function yuv2rgb (Φ (
from the scipy scipy.ndimage ndimage package. Formally, this reads
DN R.H.W B.H.H.W UDN
(21)
where
UDN Byuv algo(Usn) with algo {GF,UM)
(22)
Gamma correction (GC) equilibrates the overall brightness of the image. First, the entries of the view UDN are clipped to [0, leading to
(UCP)cл. = (VDN) a.A. (0 + ((DDN)an.
(23)
Second, the brightness adjusting transformation is defined by
PG : R.H.W [0, H, - vGc = (VCP)
(24)
for some > 0 applied elemeettwise. Note that zero-clipping is necessary for DgC to be well-defined.
In total, we define the composition
stat Proc
[0,1] H.W + [0, 1..
(25)
of the above steps
st&t Proc
&Gc o PDN 2SH o &cc PWB 4DM PBL
(26)
and call o Proc the static pipeline.
33

---

Published in Transactions on Machine Learning Research (05/2023)
ga
ga
ma, me
ma, S.
ma, U, me
ma, U, ga
ga
U, ga
bi, me
bi,
bi. me
bi.
ma, 5, me
ma, S, ga
ma, U, me
ma. U, ga
me
me.
me
Figure 8: Samples for both datasets, Raw-Microscopy and Raw-Drone, from all twelve pipelines used in the drift synthesis experiments. The legend for abbreviations can be found in Figure 3b
34

---

Published in Transactions on Machine Learning Research (05/2023)
A.1.2 Parametrized data model $ Black level correction (BL) For the parametrized black level correction define the map
BL [0, l] R RH.W (2RAW, 01) UBL. PBL. (WRAW))
(27)
and set 0 :
Demosaicing (DM) We first convert UBL to a three channel image [R, G, B] R.HH.W where the entries of R. and B are zero except
12h+112w41 BB 2011!
B2h,2 aB
22h+1,2w UBL
G2h.2w+1 2BL
parametrize
DM define the map
para [0, 1] W R333,3 R.S... (UULL, 02)
UDM
(28)
with 02 k2,, kal, where the kernels k k2, ki e R333 are separately applied to each color channel resuling in
UDM (R k1)
UDM2,.. (G k2)
DDM.. (B kzs Ns h,w
The source code of BaayeeBllnnear shows that the parameter choice
0
0.25
0
0.25
0.5
0.25
1 Kg
0.25
0.25
and k2
0.5
1
0.5
(29)
0.25
0.25
0.5
0.25
leads to
$
Bil DM
4 DM 0.02).
(30)
Towards the definition of the parameter space set O₂ R33333 x θ
White balance (WB) For the parametrized white balance define the map
S WB
12 H,W.W x 3 + R.H... (UDM, 03) owB OWW(UDD))wbrr
(31)
and set θ3 == 3 O2.
Color correction (CC) For the parametrized color correction define the map
para
R.H... R333 R.H... (uwn, 04) vec (co(VVE))MM CC
(32)
and set O₂ := 3 O
Sharpening (SH) We parametrize the sharpening filter configuration of the static pipeline, by using the entries & 3.3 defined in 17 as parameters leading to
$
para SH
133 3.H.W. 3.3 W.H..W (vCC, 0s) USH 4SH (bCC)))))
(33)
and O5 := : 3,3 04.
Denoising (DN) We parametrize the configuration where the Gaussian denoising method is applied.
Applying the Gaussian filter from scipy aaaasccciiiinmme ndimage with a = 0.5 is equivalent to convolution of the view in the YUV-color space with a specific kyaues E 55555 For the specific values kyauss see K. BLUR at the code of the pazadetrized pipeline. Therefore, to parametrize DN we define the map
4 para H.H.W 5555 3.H.W (USH, 06) VDN a PD (USH Ag
(34)
35

---

Published in Transactions on Machine Learning Research (05/2023)
and set Og := R e
Gamma correction (GC) Define the parametrized gamma correction by
S Ge R.H... R [0, 10.1133..,. (VDN. 07) v DGC @Gc
(35)
The following values were used to initialize & Proc (both "Frozen" and 'Learned") in experiment Section
5.3
Elass ParasoteiRedPoccesssan Kodule):
"""Difforoaciable proceesing pipelino via torch transtornations
Azgo:
Saeara_parasntare (tupla()ixt), optional): applisx given paraeatarx processing
track_ stages (bool, optional): shother or not to retain intereodiary stepe i0 processing
borca_aorr_ouppt: (bool, optiozal): adds Batchfora layer so the end of the procexsing ....
..init_. (colf, parrroopratttro--O0e. track, Falee batch, True): init...) self. stages Hone
11
oxper initt....i.t...
12
self beffar Kone
welf walf.track_atagee track_atagex
cancra_arraneeers DEFAULT CEMERR,
29
lovel whlte_ colour_natris canera.paraneteer
70
welf bilf.black lavel Paranaatur (torch. (black
self whife balance mn.Paraneterttorch
(ubite balance). renhaaee
self
reshapee
self- palf.aarm,ccoreee an. nr.Paraneter(tordl Tenaor (2..ar(
26
self. delf.debsyer Bebayar()
self. sharpening filter Conv2d kernel size=], padding=1, bias=Falee) wwwrphhiagg.geee...o..oooa.. weight. [0] X_3HARP clenee)
self. gausaian an. Coav2d(t, 1, kernel, sizens, paddinge2, aodeiig.. baas=Falze)
34
self self- blu wight..wagtttd [o] K. cloona)))
self bolf.baich.ug baich_aorrr Batchhoraad sffine-Falee) batch_
Tone
self. www.roogicerrbbeee. MK_00__2,2.. Y N_ ())
self ral..ragiater_bbffee ("M (K__UV_____00 22 N_VUV_2. 2..0..clone()
10
addiiadditive. laaer Hone
where
K_0
Desser ([[O,
[1 4, 1).
[o. !, o1l) /
K. torch Tensor ( [[I, 2, 11.
[2. 2)
(11)
X_B08_2_YUV torch. Teasor ([[0.299) 299, 0.687,
10
[-0.14714119,
-0.2585881e, 0.43601085],
11
60.61497638.
51486126.. 1630102611)
F. ruV 30R torch, Tensor (EL1, (tt.ooacennoee 0000000000000000
-4. -4.18277945611-83 1.1...1..11 1.1398330414++00)
[1. 00.0000000000000
-5.8042222260001 -5. 101188313877--0
11.0000000000=000
2.0820519:6264000
9825
2.0758x-04,
2.80896-05,
[2.8059
1.,
0.3
1.1582--02,
2.8089+-05)
16
02.0756e-04,
8.3731e-02, 18666---,,
8.3731e-02, 2.0765e-04
[2, 8089
1332
8.3733a-02,
1.1332e-02,
2.8089e-08;
[5, 96256-0-0
2.80396-05,
2.0756e-04,
2.8089c-05,
6.9525e-081)
K_SHARF Tensor
DEPAULT
CANERA ARPABLT_CCAMERAAC [o., O... 0., o.1.
25
[1.. I.. 1.].
[1.. o.,. o., 1., o., o., o.. 1.).
Note that the camera parameters are camera, and conversely in our case dataset, dependent and defined in the dataset classes.
36

---

Published in Transactions on Machine Learning Research (05/2023)
A.2 Description of the task models Task
Classification
Segmentation
ResNet18 based on
U-Net+++ based on 181
@ Task
trained with Adam 186 for 100 epochs
trained with Adam for 100 epochs learning rate: 7.5 10-5
learning rate: 10-4
mini-batch size: 128
mini-batch size: 12
Table 3: Summary of the training procedure for both task models.
Table 4: A set of random test samples for the segmentation task under learned processing. Top row: Targets, middle row: predictions of the task model after the first. epoch, last row: predictions the task model after the last epoch.
ResNet18 This model is designed to classify images from ImageNet 187 and has therefore an output dimension of 1000. In order to use the model to classify images from Raw-Microscopy, we changed the output dimension of the fully-connected layer to nine. The model was trained for 100 epochs using pre-trained ResNet. features. Hyperparameters were kept constant across all runs to isolate the effect of varying image processing pipelines. For implementation the code provided at httph::///// htttps://pyto:n/ http://ttootooooooooo/////oos//e org/hub/pytorch. vision, resnet/ used. The model consists of 34 layers with approximately 11.2 million trainable parameters. The storage size of the model is 44.725 MB.
U-Net++ The model was trained for 100 epochs using pretrained ResNet. features as the encoder of the U-Net+++ Hyperparameters were kept constant across all runs to isolate the effect of varying image processing pipelines. For implementation we used the code provided https://githi: wttptthu..c.m/uuuuutee.e. models. pytorch/ The model has approximately 26.1 million trainable parameters. The storage size of the model 104.315 MB.
37

---

Published in Transactions on Machine Learning Research (05/2023)
Raw training In the drift optimization experiments of Section 5.3 the raw data is demosaiced using class RawToRGB (nn. Module) from /ppocesssing/pipelliee. htcpllinnetttt_teeeeeeeeeeeeeeee py in the data model code. Then task models are tuned to raw data under the same regimes described above.
For a summary of the training procedure see Table
38

---

Published in Transactions on Machine Learning Research (05/2023)
A.3 Additional results
A.3.1
Drift synthesis
Microscopy-ISt
Microscopy-
Drone-ISP
Drone-CC
Rank
Train pipeline
Avg. score
Train pipeline
Avg. score
Train pipeline
Avg. score
Train pipeline
Avg. score
mae,
0.83
bi,u,me
0.63
magu
0,68
ma,s,ga
0.60
2
ma,u,me
0.83
me.,Sme
0.63
bi,5ga
0.68
bi.s.ga
0.57
ma,u,ga
0.82
bi,
0.62
bi,smme
0.67
me.s.pa
0.57
4
bi.s,mm
0.81
ma,,mm
0.62
ma,
0.67
mas,,me
0.55
mo
me,u,me
0.62
me,u,ga
me,s,me
6
8
0.81
0.62
me
0.67
u.ga
0.55
7
bi,a.ga
0.81
ma,u,me
0.61
ma,
0.66
bi,smme
0.34
8
me.sgga
0.80
me.sga
0.60
ma,5.g
0.66
0.54
9
me,u,me 0.80
bi,s,me
0.59
bi,u,me
0.65
me,u,me
0.53
ma,a,ga 0.80
ma.u,ga
0.59
me,s,me
0.65
me,u.ga
0.51
11
bi,u, 0.79
bi,agg
0.58
me.s.ga
0.61
bi,u,me
0.48
me, Mg 0.79
me, U,A
0.58
bl,ugga
0.61
bl,ugga
Table 5: Rankings of task models from Section 5.1 trained on different data models (columns 2, 4, 6, 8)
according to their average accuracy or IoU (columns 3, 5, 7, 9) across test pipelines respective corruptions. ISP corresponds to drift synthesis with physically faithful data models, CC corresponds to common corruptions.
Microxcooy-1SS
Rank
m0s io, me
m2 me
man sugg
man uugg
tun ture
ma. ups
m.s
mns mo
2000, 20 upus
bijaycue
mo Ruge
man taa
mns a.tt
mas moo
mus, U.
bii.. go
bl.e
max
max in
hl,t, (g
max (2
mas me
ma,n me
10a, me
ma.
ma.
mas 10
ma. me
mh me
me g2
mw.11.p m1.p
me,0,m
me 20,
bi...me
20c tue
mun u,ize
muc a,tne
100,0
bio..g
bio..g
biuu
dugg
me..gga
blogg
ma,
mo
blo..mo
blo
mo.s.ine
me.suga
meo
majo,
100,0,
bi.o.ga
10
bl.e.ee
blur.ee
blu.me
blu-pa
b1,
bl.s.ge
bi,ame
bl,a.me
bl,m,mm
12
bl,
bi.u.ge
bi.u.ga
bs,u.ga
bs,
bt,9.ga
bt,
Table 6: Ranking of task models from Section 5.1 trained under different train pipelines (rows) for each individual test pipeline (columns 2 13).
Microccopy-CC:
Idinttit
gauuee nolon
Irapu lon
speckle
gauss blir
snom
contranct
brighttmese
eaturate
elactto
bi,
bi.s.
bl.s.ga
bl,s,
bi, M...u.po
bi.
bi mn
mo....
bi.u.
b5 M. mae
me gg
bi,
mn, me
bi.i.
ma.ti. mo
bi.s,me
bi.ggg
mo m.
ma.u..e uggg
man u. ma
mis u. m.
ma.a. me
ma a.e
me..aae ane
bo.0.me
bi. me
bi, u. me
anc 2.6.
muc.a. a.c.a.am
ma. apa
mas me
bi.u.a
bi...
msc 2i
me me
hi, (1, mm
bl.s.m
bit .ps
me (24
me.- g2
ma.x-ga
mey 6, mn
ma 11.0%
max g0
bi, 10
b5.
ma.s.me
m..4. me
me, ga
bi.e.m me
me.t,
me,th me
ma,e.5
me 1..5
mm
m.... gm
maue ga
ma, m.gg
bi.u.ga
me.u.ga
bi.s.g
me ueg
ma,4,gs gg
11
b5
bi.sugo
biosuga
bi.amme
uupa blus.p
bi,amme majo.ga
mec,i,me
me.u.ga
me..u.0
mm..9.pa
mne
mne Muga
blus.ga
mo,u.ga
bl,s,me
me,o.ga
mo.,u.me
mayu,me
bl,sme
Table 7: Ranking of task models from Section 5.1 trained under different train pipelines (rows) for each individual test corruptions (columns 2 12).
39

---

Published in Transactions on Machine Learning Research (05/2023)
Drone-ISF
Rank
bi 16 mi
My mon
max My, mi
mix g
maa 14, me
m.N, 11 M.
me,6 me
mee mee.r.p gg
bica,
bi 10
bi
bt.
bs.e
biu
bi
bi
200, tum
bl.9 (5h
bi.
tne me
bi,
mayd;
bl
bl
man u, mume
mi, 11 gg
bl. gn
bl.s
mejs me
maa,
bi me
bi 15, me
max 1s, mh
mas 10, me
ma, 10, men
mas 1. me
mee m,me
me 1g
mee U. gg
me, 1. gg
me px
me,11 gg
hit hl.m.me
bt. g
mu uutae
bas u.tac
bs, u.tnc
mas i. me
bio.aa
mnc
unc 6g0
mns, dutae
bi.u.g
200, dut
200, me
10
maje. me
ma,a,me
ma,e mm
bl.e.e
ma. K.gs
me, mm
bs.e me
b5.e, me
bi,u.ga
ma, mm
11
ME.. m...
bl,t,me
m....gg ma...g
mu..
uime
bs.u,me
mea 200
bi.u,gs
bi.a.me
bi.a.me
ma.a.ga
bt,a.ga
12
bi,a.ga
bi,s,ga
bi.u.ga
me.a,ee
me.a,pe
bi.u,e
bi.u.5a
Table 8: Ranking of task models from Section 5.1 trained under different train pipelines (rows) for each individual test pipeline (columns 2 13).
Drase-CC gause blur
Rask
Identity
ICAULE nolae
abot
imppules
spackle
socm
costrant
brighttnns
saturate
elastsc
bi.
b
bl.s.g*
bl,5.5
bl.
bi,
bl.x.n
bi,
mns,
bi,amm
100,9.
Diaggg
bi,
bis
mn
u,me
bi bi.smme
incus.pa
bi, 15, me
ma..ma
blaame
m...a.m m.
maa me
ma 1 me
bi,s,ane
me.a.g
bi..mm
me mm
me.u me
cut. me
bla.tt
ma u. t.
biaame
mo 1. me
me m.
me 5.
bi,s,ane
me 10,
ts u.me
10
IS0 1. me
que. 2,0
muc 0,g0
mic u.ga
mac 0.8a
bi,
bi.s,a0
ms 2kk
bi, u.me
mac 2.
11
maje, mm
mi
b5, mm
b5 11..mmm
me,
mas (1.5%
me,ttga
b1, me
mA,3 me
12
25,10.85 gm
bi.ngg
bi. 1.,4
bd, 10.g
br,u.ga
me,u.ga
me mm
bi.
bi.
bi, 11.
Table 9: Ranking of lask models from Section 5.1 trained under different train pipelines (rows) for each individual test corruptions (columns 2 12).
ISP
Common corruptions benchmark
0.81
0.05
0.82
AXP
ABT
D3N
ABS
mu
+
3
buagge
AXT
0.62
0.72
mas.g
+
+
0.83
Repelte
STC
ABT
Microscopy
3
0.42
AB
0.6
F
8
:
1965 Ap
2
:
0.22
me
teeaage
massaga
E
:
0.48
Mentty
ganss moioe
diter
Ingosite
Specido
geuss liker
controst
brightness
catunate
edastic
Mame
bl..gn
Mlame
me
maagg
me
mesuga
meLa
Bar alpetnes
0.03
3
0.09
0.12
0.15
0:18
0.0
0.03
06
0.04
0.15
Standard
Standand rd Deviation
Figure 9: Experiment from Section 5.1 with weak severity (level 1) for the Common corruptions benchmark.
40

---

Published in Transactions on Machine Learning Research (05/2023)
ISP
Common corruptions benchmark
0.84
0.85
K.M
A43
6/1
UP
0.75
F2
Blage
0.72
0.88
ACCU DCJ
Pipel
0.45
Microscopy
0.35
0.6
+
5
:
0.25
:
0.56
mee
Te meesume
0.15
X23
0.4
Mentty
parss naloe
ditet
Irgestee
speskae
daaas Bltar
contrast
briphtnen
saturate
elastic
Bame
bcs.gn
Mame
B
ma
maraga
mm
meauga
meuun
mesugge
2ot alpelnes
0.01
3.08
0.09
0.12
0.15
0:18
0.0
0.21
0.08
0:17
0.24
0.21
Standard
Standart
Figure 10: Experiment from Section 5.1
with strong severity (level 5) for the Common corruptions benchmark.
ISP
Common corruptions benchmark 43
0.72
0.73
b.s.
ulx.me
bla.
XM2
OBS
800
000
A6B
AGY
baame
0.66
A6P
6:3
MAX
0.64
maLu
Tain ppeinm
A+1
Drone
0.58
0.58
0.43
THUS 5,00
0.52
05
messago
0.45
thot
Imsoeioa
Bansse Maar
amaurn
centrast
brigtnent
raturate
eintttc
Batme
Bldge
ingua.e
natl
met
piwliee
0.0
0.01
0.04
0.05
0.66
BGr
0.0s
0.09
0.03
0.0b
0.29
0.12
Standard Deviation
Standard Deviation
Figure 11: Experiment from Section 5.1
with weak severity (level 1) for the Common corruptions benchmark.
41

---

Published in Transactions on Machine Learning Research (05/2023)
ISP
Common corruptions benchmark
43
D.72
0.73
b.c.m
W.s.m
0.67
Blsgs
18.-99
A12
C.33
AAR
Isauue
A04
A6B
367
0.66
0.64
Tann Pe IOR
mit
THA 5-59
the
Drone
0.58
0.37
0.56
THES 5.0
0.27
0.5
05
me
0.48
liemiity
thot
Insodiar
Baise Maar
contraat
Brighteen
rnturato
elosttc
basmme
Batme
bagge
R.ge
me.igee
me.ume
meuuge
net
ppel
0.0
0.01
0.02
0.33
0.04
0.05
0.06
0.07
0.08
0.09
0.03
3.05
0.09
0.12
0.15
Standard Deviation
Standard Deviation
Figure 12: Experiment from Section 5.1
with strong severity (level 5) for the Common corruptions benchmark.
42

---

Published in Transactions on Machine Learning Research (05/2023)
ISPs
CCs
MAU.U NT
SUS,
pixel noise
Gaussian Noise
S.S.2
Contrast
mI, me
Shot Noise
blur
Brightness
Saturate
intensity
IIC. TMC
Spockle Noise
5,T11
Elastic
Excluded corruptions
Spetter
Pixelate
Defocus Blur Frosted Glass
Motion Blur
Snow
Figure 13: A comparative overview of the physically faithful data models (ISPs, top-left) and the Common Corruptions (CC, top-right) used the the drift synthesis experiments of Section 5.1 A matching heuristic based on possible visual perception the drift artifacts (lop-middle) is provided for readers who would like relate specific data models to specific corruptions. However, we emphasize that this heuristic and has metrological basis. Since CCs are not physically faithful it is not clear how relate them to actual variations in the optical data generaling process. Finally, corruptions that were excluded from the experiments in Section 5.1 are displayed (bottom). The CC examples where stitched from the original paper [88 for authenticity.
purely qualitative
43

---

Published in Transactions on Machine Learning Research (05/2023)
A.4 Datasets details
A.4.1 Data acquisition
In the following, core information on the two acquired datasets is provided. In Appendix A.4.2 you can also find detailed datasheets for both datasets, following the documentation good practices introduced by [77.
Raw-Microscopy Assossment of blood smears under light microscope is key diagnostic technique for many healthcare services such as cancer treatment and kidney failure as well as blood disorder detection 189. The creation of image datasets and machine learning models on them has received wide interest in recent years 13
191. Variations in the image processing can affect the downstream task model performance 192 Dataset drift controls can thus help to specify the perimeter of safe application for a task model. A raw dataset was collected for that purpose.
190
bright-field microscope was used to image blood smear cytopathology samples. The light source a halogen lamp equipped with a 0.55 NA condenser, and pre-centred field diaphragm unit. Filters at 450 nm, 525 nm and 620 nm were used to acquire the blue, green and red channels respectively. The condenser is followed by a 40x objective with 0.95 NA (Olympus UPLXAPO40X). Slides can be moved via piczo with 1 nm spatial resolution, in three directions. Focus was achieved by maximizing the variance of the pixel values [3] Images are acquired at 16 bit, with a 2560 x 2160 pixels CMOS sensor (PCO edge 5.5). The point-spread function (PSF) was measured to be 450 nm with 100 nm nanospheres. Mechanical drift was measured at 0.4 pixels per hour. Imaging was performed on de-identified human blood smear slides (Ma190c Lieder, J. Lieder GmbHi & Co. KG, Ludwigsburg/ Seaesbuurr/ermaayyr /Germany). All slides were taken from healthy humans without known hematologic pathology. Imaging regions were selected to contain single leukoytes in order to allow unique labelling of image patches, and regions were cropped to 256 x 256 pixcls. All images were annotated by a trained hematological cytologist using the standard scheme of normal leukocyles comprising band and segmented neutrophils, lypical and atypical lymphocytes, monocytes, eosinophils and basophils [93]. To soften class imbalance, candidates for rare normal leukocyte types were preferentially imaged, and enrich rare classes, Additionally, two classes for debris and smudge cells, as well as cells of unclear morphology were included. Labelling took place for all imaged cells from a particular smear at time, with single-cell patches shown in random order. Raw images were extracted using JetRaw Data Suite features. Blue, red and green channels are metrologically rescaled independently in intensity to simulate a standard RGB camera condition. Some pixels are discarded complementary on each channel in order to obtain a Bayer filter pattern.
Raw-Microscopy for segmentation comes with 940 raw images, twelve differently processed variants totaling 11280 images and six additional raw intensity levels lotaling 5640 samples.
Raw-Drone Automated processing of drone data has useful applications including precision agriculture 194 or environmental protection [95]. Variation in image processing has been shown to affect task model performance I 15,, underlining the need for drift controls. For the purposes of this study, raw car segmentation dataset was created for the drone image modality. A DJI Mavic 2 Pro Drone was used, equipped with a Hasselblad L1D-20c camera (Sony IMX183 sensor) having p. pixels in Bayer filter array. The lens has a focal length of mm. The f-mumber was set to N = 8, to emulate the PSF circle diameter relative to the pixel pitch and ground sampling distance (GSD) as would be found on images from high-resolution satellites. The PSF was measured to have a circle diameter of 12.5 jum. This corresponds to a diffraction- limited system, within the uncertainty dominated by the wavelength spread of the image. Images were taken at 200 ISO, a gain of 0.528 DN/e-. The 12-bit pixel values are however left-justified to 16-bits, so that the gain on the 16-bit numbers is 8.448 DN/e". . The images were taken at a hoight of 250 m, so that the GSD is 6 cm. All images were tiled in 256 x 256 patches. Segmentation masks were created to identify cars for each patch. From this mask, classification labels were generated to detect if there is a car in the image. The dataset is constituted by 548 images for the segmentation task.
Raw-Drone for segmentation comes with 548 raw images, twelve differently processed variants totaling 6576 images and six additional raw intensity levels totaling 3288 samples.
Figure 14 in A Appendix 1.4.1 provides an illustration of the imaging setup.
44

---

Published in Transactions on Machine Learning Research (05/2023)
(a)
Halogen lamp
(b)
Basophil (BAS)
Eosinoghi BEOS)
Smudge cell ISSC
Filters
Condenser
Neutruphi band (NGB)
Noubrophii seqmented (NCS)
Undiear (UC)
Sample
Objective
1-0000
4-0.05
1,=0.1
1,-0.25
1-5.5
1-0.75
1-1.0
Tube lens
Camera
Figure 14: (a) An illustration of the imaging setup. (b) Datasets visualization. (Top-leftt) RGB raw microscopy classes are shown. (Top-right) Drone raw images are shown with the segmentation mask applied over it. (Bottom) Different intensity realizations print out in the same scale of the original image. Images in the bottom row normalized on their min and max values to highlight the role of noise levels on low intensity images.
shown for the microscopy case. Tmages the top directly
Composition of Rave Mie-MMeroscopy
Class
Proportion in %
Composition of Krown
Type of iistanoss
Image and label
Dasopial (BAS)
1.91
Type of instanoss
Image and mack:
Olijatt imaga
White blood wills
Rosinelil (B06)
5.74
Objecta
Landerge slods from slozes
Type clasess
Morphologooal classes
Studgee dehrts (KSC)
17:34
Number instanoes
Number of
atypical Tymphoryte
Number original imagee
of clusses
typical Lyuphoeste (LVTT)
Imagee
Image sizer Image format
pisel
Muuscyte (MON)
20.32
pisoll
Neutrephil (hanc) (NGD)
0.53
Original mmage eise
3948 3472
Taur image format Please wes Section 6.1
Noutnehill regeemted) (NGS)
22.08
Imagy format
Image that could not be ostigeed closs (UNC)
3,19
Mask formtt
Row image fermat
DDG
Table 10: Summarics of the compositions of Raw-Microscopy and Raw-Drone
45

---

A.4.2 Datasheets
We follow the datasheets documentation framework proposed
1rr using the template https
http://ww....a....e/e/eee/aaaaaa jeap- from Christian
Garbin.
Datasheet for Raw-Microscopy
Motivation
The data set consists of 940 instances. For the pro- portion of each class in the dataset see table 11
For what purpose was the dataset crcated?
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?
With Raw-Microscopy provide publicly available raw image dataset in order to examine the effect of the image signal processing on the performance and the robustness of machine learning models. This dataset enables to study these effects for a supervised mul- ticlass classification task: the classification of white blood cells (WBCs).
The dataset does not contain all possible instances. It limited to WBC classes normally present in the peripheral blood of healthy humans. In order to cope with intrinsic class imbalance in cell distribution, rare cell class candidates such as Basophils were preferen- tially imaged.
Who created this datasct (c.g., which team, research group) and behalf of which entity (e.g., company, institution, organization)?
What data does each instance consist of? "Raw" data (e.g., unprocessed text or images) or features?
This dataset has been created by the Laboratory Applied Optics of the Micro-Nanotechuulogy group HEPIA/HES-SO,, University of Applied Sciences of Western Switzerland. Single-cell images were anno- tated by trained cytologist.
Each instance consists of an image of 256 by 256 pixels. The image is a raw image .tfff format.
Is there label or target associated with each instance?
Who funded the creation of the dataset?
The creation of the dataset has been funded by HEPIA/HES-SO.
Each instance is associated to a label, that indicates the morphological class of the image.
Is any information missing from individual in- stances?
Composition
No information missing.
What do the instances that comprise the dataset represent (c.g., documents, photos, people, countries)?
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)?
An instance is a tuple of an image and a label. The image shows a human WBCs and the label indicates the morphological class of this cell. The following eight morphological classes appear the dataset: Ba- sophil (BAS), Eosinophil (EOS), Smudge cell / debris (KSC), atypical Lymphocyte (LYA), typical Lympho- cyte (LYT) ), Monocyte (MON), Neutrophil (band) (NGB), Neutrophil (segmented) (NGS). The nith class consists of images that could not be assigned a class (UNC) during the labeling process.
No, relationships between individuals are not made explicit.
Are there recommended data splits (e.g., train- ing, cealoomenn//validattee testing)?
recommended data splits. All the data splits that used for our experiments were randomly picked.
There
How many instances are there in total (of each type, if appropriate)?
Are there any errors, sources of noise, or re- dundancies in the dataset?

---

To the best of our knowledge, there are errors in the dataset. However, key source of variability between slides from different laboratories and process- ing times is stain intensity. The samples used in this work all come from the same source, hence we assume the preanalytic treatment and staining protocol be similar. As all images were obtained on the same microscopy equipment, focus handling and illumina- tion are identical for all samples. Image labelling was performed by one trained morphologist with experi- ence in heratoloogcal routine diagnostics. It is known that morphology annotations are subject to inter- and intra-rater variability. However, we limit ourselves to normal WBCs the labeling is expected to be stable.
The donors of the blood smears used in this dataset are fully deidentified, and no information on subpipu- lation composition is provided.
Is it possible to identify individuals (i.e., one or more natural persons), either directly or in- directly (i.e., in combination with other data) from the dataset?
No. is not possible identify individuals from an image of their white blood cells or visa versa.
Does the dataset contain data that might be considered sensitive in any way (e-g., data that reveals racial or ethnic origins, sexual orien- tations, religious beliefs, political opinions union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social se- curity numbers; criminal history)?
Is the dataset self-contained, does it link to otherwise rely external resources (e.g., websites, twects, other datasets)?
The dataset is sell-contained.
No. While the distribution of cell types for a specific patient could reveal information about that patient's health status, isolated single-cell images of normal leukocytes do not allow for this inference.
Does the dataset contain data that might be considered confidential (e.g., data that is pro- tected by legal privilege or by pactor-patient confidentiality, data that includes the content of individuals non-public communications)?
Any other comments?
See table 12 for summary of the composition of Raw-Microscopy.
The dataset consist of medical data, disclosing the morphological classes of single human WBCs. In prin- ciple, the distribution of cell types conveys information on the health state of a patient.
Class
Proportion
Basophil (BAS)
1.91
However, the subjects in this datasot are fully de- ified, that the image data cannot be linked back to the healthy donors of the scanned blood smears. Furthermore, it is not disclosed which cell image was taken from which blood smear, so that no frequencies of individual cell types can be determined. Additionally, we only consider cell types present in nor- mal blood, so that no specific hematologic pathology can be deduced from cell morphologies.
Easinophil (EOS)
5.74
Sundge cell debris (KSC)
17.34
atypical Lymphoeyte (1YA)
3.19
typical Lymphocyte (LYT)
Monoryte (MON)
20.32
Neutrophil (band) (NGB)
0.85
Neutrophil (seguentedd) (NGS)
22.98
Image that could assigned (UNC) 3.19
Table 11: The proportion of the classes in Raw- Microscopy.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threat- ening, might otherwise cause anxiety?
Collection Process
How was the data associated with each instance acquired?
No. The dataset does not contain data with any of the above propertics.
Images of the dataset have been acquired directly from a CMOS imaging sensor. They are in a raw umprocessed format.
Does the dataset relate to people?
Yes. The dataset consist of images of human WBCs.
What mechanisms or procedures were used to collect the data (c.g., hardware apparatus or sensor, manual human curation, software program, software API)?
Does the dataset identify any subpopulations (e.g., by age, gender)?

---

Imaging data have been obtained via custom bright- field microscope.
Did the individuals in question consent to the collection and use of their data?
Yes, they did.
If the dataset is sample from a larger set, what was the sampling strategy (e.g., deter- ministic, probabilistic with specific sampling probabilities)?
If consent was obtained, were the consenting individuals provided with mechanism to re- voke their consent in the future or for certain uses?
Images have 256 pixel size and have been cropped from larger images. The dataset corresponds to selec- tion of white blood cells in the acquired large images. A sampling strategy aimed at increasing the propor- tion of rare classes of white blood cells has been used.
We do not know the conditions of consent adopted by the selling company. However, we believe the com- pany provided the individuals a complete freedom revoking their consent in the future, if desired.
Who was involved in the data collection process (e.g,, students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted?
No, this kind of analysis has not been conducted.
A research assistant has been involved in the data collection process and has been compensated with a monthly salary.
Prepoceesing/ddennnngg/abliiii
Over what timeframe was the data collected? Does this timeframe match the creation time- frame of the data associated with the instances (e.g,, recent crawl of old news articles)?
Was any preproeesinn/ddaannng///aaal of the data done (e.g., discretization or bucketing, to- kenization, part-of-speech tagging, SIFT fea- ture extraction, removal of instances, process- ing of missing values)?
Data have been collected on timeframe of two months, corresponding the availability of the phys- ieal samples to image. Data have been collected on purpose for this work.
Intensity scaled images generated with Jetraw Data Suite for both datasets, which applies a physi- cal model based on sensor calibration to accurately simulate intensity reduction. Microscopy Raw images are extracted from RGB Microscopy data through a pixel selection from images taken with three filters, in order to have Bayer Pattern. Pixels intensities rescaled with Jetraw Data Suite to match the mea- sured transmissivities of Bayer colour filters array.
Were any ethical review processes conducted (e.g., by an institutional review board)?
The microscopy data was purchased from a commer- cial lab vendor (J. Lieder GmbH & Co. KG, Lud- wigsburg/ /eruuayy) who attained cousent from the subjects included.
Does the dataset relate people?
Was the "raw" data saved in addition to the rreprocessed/cleaneee releaendd/lbbeeeeddd /labeled data (e.g., to support unanticipated future uses)?
Yes. The dataset consists of microscopic images of human white blood cells.
Raw images are available in the dataset.
Did you collect the data from the individuals in question directly, or obtain it via third parties other sources (e.g., websites)?
Is the software used to preproces/cccan//abl the instances available?
Data have not been obtained via third partics.
All code used in the experiments of this manuscript is publicly available. Jetraw products that were used for acquiring the data are commercially available.
Were the individuals in question notified about the data collection?
As the blood smear slides were bought from com- pany, notification to individuals of the data collection has been performed by the company.
Uses

---

Has the dataset been used for any tasks al- ready?
Will the dataset be distributed under copy- right or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?
The dataset has not yet been used.
The dataset will be distributed under the Creative Commons Attribution 4.0 International.
Is there a repository that links to any or all papers or systems that use the dataset?
Have any third parties imposed IP-based or other restrictions on the data associated with the instances?
The repository at https://githini com/ Seaudit-org/ra2llogg associated this work, maintained by Luis Oala.
No.
What (other) tasks could the dataset be used for?
Do any export controls or other regulatory re- strictions apply the dataset or individual instances?
The dataset can be used to study the effect of image signal processing on the performance and robustness of any other machine learing model implemented in PyTorch, designed for a supervised multiclass classifi- cation task.
There such restrictions.
Maintenance
Is there anything about the composition of the dataset or the way it was collected and pre- processel/leenne//bbellle that might impact. future uses?
Who will be supportinghhattnn/mmaaaaauuig the dataset?
Luis Oala on behalf of Dotphoton AG.
To the best of our knowledge, we do not recognize
How can the owner/curator/ mannagrr of the dataset be contacted (e.g., email address)?
such impacts.
Are there tasks for which the dataset should not be used?
By email address via luis www..aaladoophhoooocom
To the best of our knowledge, there are no such tasks.
Is there an erratum?
At the time of submission, there is no such erra- tum. If an erratum needed the future will be accessible htttss///github... wttp://aa./aaaaaaooooooaaoo raw2logit
Distribution
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset. was created?
Will the dataset be updated (e-g., correct labeling errors, add new instances, delete in- stances)?
Yes. The datasct will be enlarged wrt. the number of instances.
Yes. The dataset will be publicly available.
How will the dataset will be distributed (e.g., tarball website, API, GitHub)
If the dataset relates to people, are there ap- plicable limits on the retention of the data associated with the instances (e.g., were indi- viduals in question told that their data would be retained for a fixed period of time and then delcted)?
A guide to access the datasot is available at https: Moreover, the dataset can be downloaded anonymously and di- rectly at hattps://eenod..oet http:///wwdo..rrrrerrrro/////a5/ un-
com wtt.uu..comaiiaadd-orrrrraaaaa/a
der the doi: 10.5281 10..228//7enncoo55235553
To the best of our knowledge, there such limits.
When will the datasct be distributed?
Will older versions of the dataset continue to be supported hwwpe///hosotttttttttaiiaaa
The dataset is already publicly available.

---

Older versions will be supported and maintained in the future. The dataset will continue to be hosted as long as https://zenodo. org/ exists.
For any of these requests contact either Luis Oala (niisoolladdoophoooon)) Bruno Sanguinetti wwr.u..snguiieiiiioooooooo.ooom
For now, we do not have an established mechanism to handle these requests.
If others want to (atend/augmentt /buld on/- contribute to the dataset, is there a mechanism for them to so?
Composition of Raw-Microscopy
Type of instances
Image and label
Objects on images
White blood cells
Type of classes
Morphological classes
Number of instances
940
Number of classes
9
Image size
256 by 256 pixels
Image format
.tff
Raw image format Please see Section 4.1
Table 12: A summary of the composition of Raw- Microscopy.

---

Datasheet for Raw-Drone
Motivation
Does the dataset contain all possible instances or is it sample (not necessarily random) of instances from larger set?
For what purpose was the dataset ercated?
The dataset does not contain all possible instances. Only images with at least one white pixel in the asso- ciated segmentation mask are considered.
With Raw-Drone provide publicly available raw dataset in order to examine the effect of the image data processing on the performance and the robust- ness of machine learning models. This dataset enables to study these effects for a segmentation task: the segmentation cars. The dataset was taken with specified parameters: sensor gain, point-spread func- tion and ground-sampling distance, so that physical models may be used to process the data. It also laken with a easily accessible and affordable system, so that it may be reproduced.
What data does each instance consist of? "Raw" data (e.g., unprocessed text or images) features?
Both, the image and the segmentation mask consist of 256 by 256 pixels. The image is a raw image in tiif format and the the segmentation mask is in .nng format. The images are cropped sub-images of 12 raw images in DNG format, consisting of 3648 by 5472 pixcls.
Who created this dataset (c.g., which team, research group) and behalf of which entity (e.g., company, institution, organization)?
Is there label or target associated with each instance?
The dataset was created by Bruno Sanguinetti and Marco Aversa on behalf of the company Dotphoton AG.
Each instance associated to a binary segmentation mask.
Is any information missing from individual in- stances?
Who funded the creation of the dataset?
The data collection was funded by Dotphoton AG. The calibration of the image characteristics was jointly funded by Dotphoton AG and the European Space Agency.
No information is missing.
Are relationships between individual instances made explicit (e-g., users' movie ratings, social network links)?
Since every image is a cropped sub-image of an origi- nal image, several of these sub-images belong to the same original image. All sub-images are disjoint, i.e. no different images share a pixel from the original image.
Composition
What do the instances that comprise the dataset. represent (e.g., documents, photos, people, countries)?
Are there recommended data splits (e.g., train- ing, caalooment/valldattinnn
testing)?
An instance is tuple of an image and a segmentation mask. The image shows a landscape shot. from above. The sogmentation mask is a binary image. white pixel this mask corresponds to a pixel within a region in the image where car is displayed. A black pixel in this mask corresponds to pixel within region in the image where no car is displayed.
There are no recommended data splits. All the data splits that we used for our experiments were randomly picked.
Are there any errors, sources of noise, or dundancies in the dataset?
To the best of our knowledge, there are no errors in the dataset. The segmentation mask created by hand and hence noisy, especially at the boundaries between region with car and region without a
How many instances are there in total (of each type, if appropriate)?
The dataset consists of 548 instances.

---

Is the dataset self-contained, does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? The dataset is self-contained.
objective has focal length of 10.3 mm. We operated this objective at f-number of N 8, to emulate the PSF circle diameter relative to the pixel pitch and ground sampling distance (GSD) as would be found on images from high-resolution satellites. Oper- ating at N 8 also minimises vignetting, aberrations, and increases depth of focus. The point-spread func- tion (PSF) of 12.5 pm using the edge-spread function technique and a ground calibration target. This corresponds to a = 2.52 px, which also corresponds to a diffraction- limited system, within the uncertainty dictated by the wavelength spread of the image. Images were taken 200 ISO, corresponding to gain of 0.528 DN/e". The 1it pixel values are however left-justified to 16-bits, that the gain on the 16-bit nunbers is 8.448 DN/e. The images were taken at height of 250 m, so that the GSD is 6em. All images were tiled 256x256 patches. Segmentation color masks were created to identify cars for each patch. From this mask, classification labels were generated to detect if there is in the image. The dataset is constituted by 548 images for the segmentation task, and 930 for classification. Six additional intensity scales were created with Jetraw.
Does the dataset contain data that might be considered confidential (e.g., data that is pro- tected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals non-public communications)?
measured to have circle diameter
No. The dataset does not contain data of any of the above types.
Does the data set contain data that, if viewed directly, might be offensive, insulting, threat- ening, might otherwise cause anxiety?
No. The dataset does not contain data with any of the above properties.
Does the dataset relate to people?
The dataset does not relate to people. The drone data was screened for PITs such as faces or license plates on cars and removed by the data collection team.
Any other comments?
If the dataset is sample from a larger set, what was the sampling strategy (e.g., deter- ministic, probabilistic with specific sampling probabilities)?
See table 13 for summary of the composition of the Raw-Drone,
The entire dataset is presented.
Collection Process
Who was involved in the data collection process (c.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
How was the data associated with each instance acquired?
The data was collected by flying a drone and saving the raw data. The calibration data for the drone's imager acquired both under laboratory conditions and using ground- bosedd-based calibration target, so that it could be acquired under operating conditions.
The datasct was taken by a company employee, com- pensated by his salary. Labeling was performed by both company employee and PhD student, who's PhD funded by the company.
Over what timeframe was the data collected? Does this timeframe match the creation time- frame of the data associated with the instances (e.g., recent crawl of old news articles)?
What mechanisms or procedures were used collect the data (e-g., hardware apparatus or scnsor, manual human curation, software program, software API)?
The dataset was taken as the initial step of writing this article.
To acquire the drone images, used DJ1 Mavic 2 Pro Drone, equipped with Hasselblad L1D-20c cam- era (Sony IMX183 sensor). This system has 2.4 um pixels in Bayer filter array. Images were taken with the drone hovering for maximum stability. This sta- bility was verified to be better than a single pixel by calculating the correlation of subsequent images. The
Were any ethical review processes conducted (e.g., by an institutional review board)?
The dataset does not contain any elements requiring ethical review process.

---

Does the dataset relate to people?
To the best of our knowledge, we do not recognize such impacts.
The dataset does not relate to people. There are indi- viduals on the images, but it is not possible to identify these individuals.
Are there tasks for which the datasct should not be used?
To the best of our knowledge, there are no such tasks,
Preprocesinggdaaaning//bbbbigg
Was any preppocesing/ddlaaing///bbee
the data done (e.g,, discretization or bucketing, to- kenization, part-of-specch tagging, SIFT fea- ture extraction, removal of instances, process- ing of missing values)?
Distribution
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?
No further processing was applied to the Raw-Drone data.
Yes. The dataset will be publicly available.
Was the "raw" data saved in addition to the celroossssed/ceaeeedeeee /labeled data (e.g., to support unanticipated future uscs)?
How will the dataset will be distributed (e-g., tarball on website, API, GitHub)
A guide to access the dataset is available at https //github. com/ /tt///addttto///aallootttt Moreover, the datasct can be downloaded anonymously and di- rectly at
Raw images are available in the dataset.
Is the software used to preprocess/clean/laee the instances available?
un-
der the doi: 1.0.5281/zee /2221//222ood.55353333.
All code used in the experiments of this manuscript is publicly available. Jetraw products that were used for acquiring the data are commercially available.
When will the dataset be distributed?
The dalaset is already publicly available.
Uses
Will the dataset be distributed under a copy- right or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?
Has the dataset been used for any tasks al- ready? The dataset has not yet been used. The datasct will be distributed under the Creative Commons Attribution 4.0 International.
Is there a repository that links to any or all papers or systems that use the dataset?
Have any third parties imposed IP-based or other restrictions on the data associated with the instances?
The repository at https://gitais com/ Staudit-org/ra22loo associated to this work, maintained by Luis Oala.
No.
Do any export controls or other regulatory re- strictions apply to the dataset to individual instances?
What (other) tasks could the dataset be used for?
The dataset can be used to study the effect of image signal processing on the performance and robustness of any other machine learing model implemented in PyTorch, designed segmentation task.
There are no such restrictions.
Maintenance
Is there anything about the composition of the dataset or the way it was collected and pre- processed/cleane /censed//eenee//lbbeeed that might impact future uses?
Who will be suppottigglostingnaaaattiiiii the datasct?
Luis Oala behalf of Dotphoton AG.

---

How the twner/curate paattorrmmaagerr of the dataset be contacted (e.g., email address)? By email address via luis.olla aotp@potpoooonccom
viduals in question told that their data would be retained for fixed period of time and then deleted)?
To the best of our knowledge, there are no such limits.
Is there an erratum?
Will older versions of the dataset continue to be supported/ hosted/////////// musted/mminnaiieed: maintained?
At the time of submisson, there is no such erra- tum. If an erratum is needed in the future it will be accessible at https://giihhb..cm com/a raw2logit
wttpt/tu...ommaauui/aiaaaaaaaaaa aiaudit-org/
Older versions will be supported and maintained in the future. The dalaset will continue to be hosted as long as https://zenodo org/ exists.
Will the datasct be updated (c.g., correct. labeling errors, add new instances, delete in- stances)?
If others want to extend /augmeet, /build on/- contribute to the dataset, is there mechanism for them to do so?
Yes. The datasct will be enlarged wrt. the number of instances.
For any of these requests contact either Luis Oala [uiis.onla@kooophotoon.ooo www.eeaaiiiieii@ooooooooooooooo.
or Bruno Sanguinetti
the dataset relates to people, are there ap- plicable limits on the retention of the data associated with the instances (e.g., were indi-
For now, we do not have an established mechanism handle these requests.
Composition of Raw-Drone
Type of instances
Image and mask
Objects images
Landscape shots from above
Number of instances
Number of original images
12
Image size
256 by 256 pixels
Mask size
by 256 pixels
Original image size
3648 by 5472
Image format
tif .pnn .DNG
Mask format
Raw image format
Table 13: A summary of the composition of Raw-
Drone.


Reviews:
[{"review_id": "I9GdvMHFEV", "reviewer": "Reviewer_Fgxr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The paper introduces a novel and interesting way to think about the preprocessing in computer vision problems that allows for a more realistic (but limited) characterisation of data drift.\n- The authors collect and release novel datasets allowing the study of data preprocessing pipelines on raw images.\n- The paper provides an interesting comparison of the data drifts induced by the proposed data models as well as more commonly used data augmentations.\n\nCons:\n\n- In the current form of the paper, it is not clear in which scenarios one would use a data model as proposed. The authors show three different use cases in terms of 1) \"drift analysis\", 2) \"drift forensics\" and 3) \"drift optimisation\". However, it is not clear how useful those applications are in practice.\n  - 1) assumes that one would build a data model to predict the difference between sites `s` and `t` from only using data acquired on site `s`. However, this assumes that the data model for `t` is fixed and one does not have access to the raw data. Furthermore, it is only really useful for characterising the changes due to the data model and does not capture changes due to the optics or other drifts.\n  - 2) assumes that it is interesting to understand a models vulnerabilities due to changes in  the data preprocessing. This is only relevant if the model does not \"contain\" the data processing pipeline as well. But the paper seems to argue that raw data should always be considered when modelling a task.\n  - 3) This is potentially the most useful one. However, it is possible to also optimise the processing of \"preprocessed data\" e.g. the parameters of the gamma correction. Furthermore, it would then lack comparisons to works optimising the preprocessing such as [1].\n- The data model is currently very rigid in it's form. It would be interesting to expand upon this by considering different orders of processing steps. It also is not clear where this exact pipeline comes from and how it relates to common image processing pipelines implemented by, e.g. camera manufacturers.\n\nMinor:\n- I'd suggest changing the title to make it clear this work is limited to \"optical images\".\n- Introduction: It is said the gradient of the data model allows for \"precise tolerancing\" of the task model sensitivity. What's meant by that?\n- What's eq. 1 supposed to convey? I'm not sure how to read the equation.\n- Sec 1.1. explains the need for physically realistic data models. However, I'd argue that public APIs might be exposed to changes that are not physically reasonable. Furthermore, it lacks any mention of potential changes to $x_{raw}$ due to changes in optics or image acquisition or otherwise. E.g. the drone dataset could be exposed to data drift because of seasonal changes to the environment.\n- Sec 1.1. - it is argued that adding Gaussian noise is not physically realistic. However, could this be used for modelling sensor noise either by adding it to $x_{raw}$ or the processed image.\n- The contribution mention that the drone dataset is a regression dataset - however it seems to be a segmentation one?\n- How are data models etc related to data normalisation?\n- Sec 4.1. mentions that the datasets are collected keeping \"positive impact\" in mind. However, I'd be careful with the drone dataset.\n- Sec 4.1. assumes that images are always RGB images. How could this be extended to greyscale images or images where multiple \"channels\" are captured with specific colour filters? (This isn't uncommon in biology or potentially astronomy or others)\n- Sec 4.1. - I believe it might be clearer to say \"The lens has a focal length of ...\" instead of \"objective\".\n- Sec. 4.1. - What are segmentation color masks?\n- Fig 3. could use a general caption.\n- Sec 4.2.1 \"in 3a\" should be \"in Fig. 3a\"\n- Why is color correction performed in YUV space? Is the color conversion differentiable?\n- Why do you define the parameter space sets for every operation?\n- Fig 4: How come the diagonal isn't best for a given data model?\n- Fig 4: There seems to be a mismatch with the \"*\". ma,u,me is the best on \"identity\" on the right, but not the best for \"ma,u,me\" test pipeline.\n- Fig 4: How is the worst case defined?\n- Fig 4: How do you take the average when saying \"the average performance ... drops ...\"?\n\n- Fig 6.c: What's the forensics loss?\n- Sec 5.2.3: Why would the data model not be part of the \"shipped model\"?\n- Fig 7: What does \"Raw comparison\" refer to? Is this the model trained on raw data? If so, it should  be integrated into the the other plots so that it is easy to compare. How is the raw model trained? For segmentation, does the target segmentation mask need to be processed to fit the raw data space before demosaicing?\n- Fig 7: It is said this isn't possible to run similar optimisation on processed data. However, it would be possible to tune hyperparameters of gamma correction or similar. Furthermore, there is work on learning data normalisation in medical imaging [1]\n\n[1] Drozdzal, Michal, et al. \"Learning normalized inputs for iterative estimation in medical image segmentation.\" Medical image analysis 44 (2018): 1-13.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes the use of explicit data models for the processing of raw (optical) imaging data to 1) enable insights into the performance of a \"task model\" under \"more realistic\" data drift; 2) the analysis of vulnerabilities of the task model in  terms of data processing; 3) the joint optimisation of the data processing pipeline together with the task model.\n\nI believe the usage of explicit modelling of the data processing pipeline to be novel and interesting to the community. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please expand on the way the data models would be used in practice and what requirements go hand-in-hand with these scenarios.", "broader_impact_concerns": "Nothing major.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MVBMKfJfbe", "reviewer": "Reviewer_R3iq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The requested changes by the AE were partially addressed and the paper is improved. However, some issues remain unaddressed. Overall it would be helpful if all changes over the last revision were highlighted for ease of reference.\n\nSuggestion 1: \n- I may have missed something, but it did not appear to me there were any changes to the introduction (highlighted in blue), beyond the caption of Figure 1. \n- It is good that a separate section discussing potential use cases and limitations is included for each of the three applications. However, it still remains unclear what the value of modelling the image processing pipeline is for drift forensics if RAW data is avaliable - any shifts due to image processing can be avoided by training on the RAW data; modelling the image processing pipeline will not help for shifts due to the sensor or optics, which are not captured in the pipeline. Similarly, the motivation for drift synthesis is again not clear if RAW data is available at both sites. The exact setup and use-case should be further clarified.\n\nSuggestion 2: \n- It does not appear that additional experiments were included beyond those included in the revision for the original submission as Appendix A.4.3; this result has been moved into the main text, which is good. However, more extensive evaluation should be done on the RAW data - for instance, at least 3 cross-validation runs per dataset, and task models should be tuned for the RAW data; details on the RAW data training should be included in A.2. The caption for Figure 7 should also be updated accordingly.\n- I did not see an expanded discussion on RAW training in Section 6.\n\nSuggestion 3:\n- The changes to Section 5.1 address my concerns. One finer point which would improve the paper would be to consider the actual impact of incorrect model selection on performance, by comparing the difference in performance on the shifted data when using the best model selecting using physically-faithful and post-hoc corruption cases.\n- On inverting processing - the point is that if the reconstructed RAW data is sufficiently good for training models that generalize, then modelling the image processing pipeline may not be necessary. As such it is a relevant comparison and should be at least discussed in the paper.\n\nSuggestion 4:\n- While the processing may be done in silico, there will likely be sensor noise which has to be accounted for in any real application. Using real data will give a sense of this.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a revision of a previously reviewed submission. Briefly, it proposes modelling the image processing pipeline in imaging devices to help with simulating and recovering from domain shifts due to changes in the image processing.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify the use-cases and motivation in the setup where RAW data is available\n- Provide more comprehensive experiments on RAW data for section 5.3 - multiple runs and model tuning for RAW data\n- Include a discussion and if possible, comparison using reconstructed RAW data\n- [Optional] Analyze the effect of incorrect model selection on actual task performance for section 5.1\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dj8wzZOLf0", "reviewer": "Reviewer_XUNK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The proposed framework is novel and under-explored.\n- Data model optimization part is well supported by the experiments.\n\nWeakness\n- It is unclear why we should compare physically faithful one and physically non-faithful one as side by side in Figure 4 and 5.\n- The paper length can be significantly reduced (and make it more concise).\n- Limitation and assumptions should be clearly clarified and should be mentioned much earlier.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors propose a new framework for dataset drift for multiple use-cases: robustness evaluation, data model optimization, etc.\n- The authors show some interesting experimental results on both robustness evaluation and data model optimization. Especially, the data model optimization is interesting that it can further improve the downstream performance significantly.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Figure 1\n- It would be great if the authors can explain more about drift synthesis, drift forensics and drift optimization.\n- The explanation of those three things (in page 4) is far from the Figure. Good to add some brief summaries.\n\n2. Figure 4 and Figure 5\n- What is the meaning of comparing ISP and common corruption benchmarks?\n- The conclusion for each left and right figure is different. Left figure shows the robustness of the physically faithful one and the right figure shows the robustness of the physically non-faithful one.\n- When we test the robustness, we may check both parts. For instance, brightness can commonly occur in dataset shifts and it would be also important to make a robust model on the brightness. \n- Also, blur is another important part in the Autonomous vehicle case (imagine the rainy day). \n- So, it is unclear why the authors compare those two as side by side even though those are not apple to apple comparisons.\n\n3. Limitations & Assumption\n- As we know, the authors assume that we have the raw sensor data. However, this is not a common setting and it is very hard to find the public datasets with raw sensor data.\n- It would be good if the authors state this assumption in the earlier part of the paper (much earlier than page 6).\n\n4. Paper layout\n- Personally, some parts of the paper can be moved to the appendix to make this paper more concise. \n- For instance, the details of Section 4.1 can be moved to the appendix.\n- Majority of Section 4.2.1 and 4.2.2 can be moved to the appendix.\n\n5. Etc\n- Figure 2: Microscopy should be columns one to two.\n- It would be great if the authors use the consistent term in the paper. For instance, in some cases, the authors use \"regression / segmentation\" interchangeably. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript is improved considerably compare to the last submission and it improved even more during the discussion period. Authors have improved the writing including the discussions on the limitations of the proposed technique. Furthermore, the main missing baseline which was training on raw data has been added to the paper. After the discussion period, all reviewers recommended acceptance. Based on all above, my final recommendation is to accept the paper as is.
Decision: Accept (poster)