Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Luis Oala * 1 2 Marco Aversa * 2 3 Gabriel Nobis 1 Kurt Willis 1 Yoan Neuenschwander 4 Michèle Buck 5
Christian Matek 6 Jérôme Extermann 4 Enrico Pomarico 4 Wojciech Samek 1 Roderick Murray-Smith 3
Christoph Clausen 2 Bruno Sanguinetti 2

Abstract
This study addresses robustness concerns in machine learning due to dataset drift by integrating physical optics with machine learning to create explicit, differentiable data models. Thesemodels illuminate the impact of data generation on model performance and facilitate drift
synthesis, precise tolerancing of model sensitivity (drift forensics), and beneficial drift creation (drift optimization). Accompanying thestudy are two datasets, Raw-Microscopy and RawDrone, available at https://github.com/
aiaudit-org/raw2logit.

Camera image data has played a crucial role in advancing the field of machine learning and also features heavily
in important application domains including medicine andgeospatial modeling. Unfortunately, machine learning systems can fail depending on their inputs, making robustness
essential to handle variations in input data [88; 75; 83].
This study illustrates the use of explicit data models forimages to manage dataset drift in machine learning workflows, facilitating the creation of reliable validation protocols for critical domains. We define dataset drift through(X RAW , Y ) ∶ Ω → RH,W × Y, the raw sensor data generating random variable on a probability space (Ω, F, P). Raw
inputs xRAW undergo a data model ΦProc ∶ RH,W → RC,H,W ,providing a processed view v = ΦProc (xRAW ) for training
a task model ΦTask ∶ RC,H,W → Y. A different data modelΦ̃Proc creates a new view Ṽ = Φ̃Proc (X RAW ) of X RAW ,inducing dataset drift
Ds = P ○ (Ṽ , Y )−1 =/ Dt .

(1)

Dataset drift can stem from variations in camera types or set*

Equal contribution 1 Fraunhofer HHI, Berlin, Germany
Dotphoton AG, Zug, Switzerland 3 University of Glasgow, Glasgow, United Kingdom 4 HEPIA/HES-SO, Geneva, Switzerland
5
Klinikum rechts der Isar, Munich, Germany 6 Helmholtz Zentrum Munich, Munich, Germany. Correspondence to: Luis Oala<luis.oala@dotphoton.com>.
2

Published at the Differentiable Almost Everything Workshop of the
40 th International Conference on Machine Learning, Honolulu,
Hawaii, USA. July 2023. Copyright 2023 by the author(s).

tings. Robustness validation is not just an engineering task
but also mandated by quality standards [93; 27; 61]. Failure
to do so has obstructed the application of machine learning technology in impactful fields [8; 85; 23; 52]. Hence,
real-world robustness validation of image machine learning
systems is not just an intellectual exercise but a necessity
for successfully integrating machine learning research withreal-world infrastructures and data. Image dataset drift validation employs either augmentation testing, which applies
perturbations like Gaussian noise to images [31; 16; 54], or
catalogue testing, which collects diverse camera datasets[40; 2; 55; 45]. Augmentation can produce unfaithful drift
artifacts, limiting its physical accuracy [90; 96; 35]. Conversely, catalogue testing assures physically faithful samples
but requires extensive data collection. The data models ofimages have received minimal focus in machine learning
robustness studies, often treated as a black-box despite theirimportance [77]. This neglect is surprising, as explicit data
models are crucial in optics, metrology, and industry applications [7; 73; 89; 13; 28; 64; 101; 99; 38; 74]. For acomprehensive taxonomy of related work please refer to Appendix B.4. We bridge machine learning and physical optics
to generate explicit, differentiable data models for flexible,
physically faithful drift controls. Our primary contributions
are 1 :
1 Drift synthesis: Physically faithful drift test case synthesis (Section 2.1). 2 Drift forensics: Gradient propagation from the task model to the data model for precise
data forensics (Section 2.2). 3 Drift optimization: Gradient connection adjustment between data and task model for
better model performance (Section 2.3).

1. Methods
Advanced image data models necessitate raw sensor data,
common in fields like microscopy, biomedicine, and autonomous vehicles. Modern digital camera systems, including smartphones, provide access to this data. We explain its
procurement from optical hardware and share two datasets
1
This workshop manuscript highlights the most important components of our data models approach. For the full paper pleasesee our publication in the Transactions on Machine Learning
Research [60] at https://openreview.net/forum?id=
I4IkGmgFJz.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Figure 1: Illustration of an optical imaging pipeline and raw-enabled drift controls. The measurement process outputs raw data xRAW which undergoes image signal
processing (ISP) ΦProc . The processed data is consumed by a machine learning task model ΦTask that outputs ŷ. Combining raw data, machine learning pipeline, and adifferentiable data model allows drift controls: 1 drift synthesis (creation of faithful drift test cases), 2 drift forensics (specifying data environments to avoid), and 3 drift
optimization (using task model gradient to optimize data generation).

for machine learning tasks at https://zenodo.org/
record/5235536.
1.1. Raw dataset acquisition
We introduce two raw datasets, Raw-Microscopy and RawDrone, to fill the gap of calibrated, labelled raw data. RawMicroscopy contains annotated blood smear microscope
images, and Raw-Drone includes annotated drone car images. These diverse datasets representing classification andregression tasks underscore the importance of robustness
and drift controls in high-stakes scenarios. The datasets’details are in Appendix B.5.1 and Figure 13 [24].
1.2. Data models: Image signal processing ΦProc
Image data transitions from raw state xRAW to processed
image v through image signal processing ΦProc ∶ RH,W →RC,H,W [73]. Raw sensor images differ from typical machine learning input due to transformations such as correction, denoising, and sharpening that produce an RGB imagev [7; 44; 26; 92]. These transformations result in varied images contributing to dataset drift, as shown in visual abstract
on page 1.
Let (X RAW , Y ) ∶ Ω → RH,W × Y be the raw sensor data generating random variable on some probability
space (Ω, F, P), with Y = {0, 1}K for classification and
Y = {0, 1}H,W for segmentation. Let ΦTask ∶ RC,H,W → Y

be the task model determined during training. The inputs
that are given to the task model ΦTask are the outputs of thedata model ΦProc . We distinguish between the raw sensor
image xRAW and a view v = ΦProc (xRAW ) of this image,where ΦProc ∶ RH,W → RC,H,W models the transformation
steps applied to the raw sensor image during processing.
The objective in supervised machine learning is to learn atask model ΦTask ∶ RC,H,W → Y within a fixed class of task
models H that minimizes the expected loss wrt. the loss
function L ∶ Y × Y → [0, ∞), that is to find Φ⋆Task such thatinf E[L (ΦTask (V ), Y )](2)
ΦTask ∈His attained. Towards that goal, ΦTask is determined during
training such that the empirical error
1 N(3)∑ L (ΦTask (v n ), yn )N n=1
is minimized over a sample S = ((v 1 , y1 ), ..., (v N , yN ))of views. Modelling in the conventional machine learning setting begins with the image data generating random variable (V , Y ) = (ΦProc (X RAW ), Y ) and the target distribution Dt = P ○ (V , Y )−1 . Given a dataset drift
Ds = P ○ (Ṽ , Y )−1 =/ Dt , as specified in Equation (1),without a data model we have little recourse to disentangle
reasons for performance drops in ΦTask . To alleviate thisunderspecification, an explicit data model is needed. We
consider two such models in this study: a static model ΦstatProc
and a parametrized model Φpara
Proc . Following common steps

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

in ISP, the static data model is defined as the compositionΦstat
Proc = ΦGC ○ ΦDN ○ ΦSH ○ ΦCC ○ ΦWB ○ ΦDM ○ ΦBL , (4)mapping a raw sensor image to a RGB image. We notethat other data model variations, for example by reordering
or adding steps, are feasible. The static data models allow the controlled synthesis of different, physically faithfulviews from the same underlying raw sensor data by manually changing the configurations of the intermediate steps.
Fixing the continuous features, but varying ΦDM , ΦSH andΦDN results in twelve different views for the configurations
considered here. The parametrized data model, Φpara
Proc , maps
from a parameter space, Θ, to an RGB image from a raw
sensor image. Each processing step of this model is differentiable with respect to parameters θ, enabling backpropagation of the gradient through the model for drift forensics andadjustments. This data model is differentiable in θ, fulfilling
para
stat
)
(5)
Φstat
Proc = ΦProc (⋅, θ
stat
for a specific parameter set θ and static pipeline configuration Φstat
Proc . Using the components specified earlier, theparametrized processing model is defined as
3,H,W
Φpara
×Θ → [0, 1]3,H,W , (xRAW , θ) ↦ v (6)Proc ∶ [0, 1]
by composing:
para

para

para

1 Drift synthesis with Φstat
Proc : Microscopy

Figure 2: The figure depicts 5-fold cross-validation results for Raw-Microscopy
drift synthesis experiments. Each cell indicates the average accuracy with a standard
deviation border. Task models were trained on vertical axis data models and testedon the horizontal axis processed data. Numbers 1-3 left to the vertical axis rank task
models by average accuracy across all test pipelines, with stars marking the training
pipeline with the best performance for each test pipeline/corruption. Full ranking
results are available in Appendix B.3. Top-left: Data model variations cause mild
performance drops. Top-right: Comparison to a corruption benchmark shows a 13x
higher average performance drop. Bottom: Visual inspection of worst-case train/test
pipelines.

̃ ∥2 − L(Ṽ,Y )2 Drift forensics with ∇θ λ∥V − V2parav = (ΦGC (⋅, θ7 ) ○ ΦDN (⋅, θ6 ) ○ ΦSH (⋅, θ5 ) ○ ΦCC (⋅, θ4 ) ○
para

para

para

ΦWB (⋅, θ3 ) ○ ΦDM (⋅, θ2 ) ○ ΦBL (⋅, θ1 ) (xRAW ) .

(7)

The operations used above are differentiable except for theclipping operation in the GC that is a.e.-differentiable2 , if
the set {0, 1} of non-differentiable points has measure zero.
Hence, assuming that P ((vDN )c,h,w ∈ {0, 1}) = 0 holds
true for the entries of v DN results in an a.e.-differentiable
processing model. We further say that Φpara
Proc is differentiable,
noting that this holds only a.e. under the aforementioned
assumption.

2. Applications
With data models, raw data and task models in place we are
now able to demonstrate the advanced dataset drift controls
comprising 1 drift synthesis, 2 modular drift forensics
and 3 drift optimization.
2.1. Drift synthesis
The static data model enables controlled synthesis of processed views from a single raw dataset, aiding in model
validation against device-specific drift. In our experiments,
we used twelve data models, training task models on one
and testing them on the other eleven. Results are meanvalues over 5-fold cross-validation (see Appendix B.2 forsetup details). The leukocyte classification model, shownin Figure 2, demonstrates robustness to processing-induced
drift, except for the (ma,s,me) configuration. The segmenta2

a.e. stands for almost everywhere

Figure 3: (a): The figure shows test accuracy on the Raw-Microscopy test set
after 20 epochs of adversarial search in the data model for different regularization
weights λ. Left plot represents various pipeline parameter selections. Right plot̃ para anddisplays ℓ2 -norm (of processed images between the adversarially trained ΦProc
the default Φpara
Proc ) versus attained task model accuracy. (b): Processed samples
from the drift forensics after 20 epochs with various regularization weights λ. (c):Similar results for Raw-Drone. Lower regularization results in a larger adversarial
optimization search space. Forensics loss refers to the binary cross entropy and Diceloss used for the segmentation task model.

tion model (Figure 7) shows varied performance across data
model combinations, with average performance droppingfrom 0.82 to 0.8 for classification and from 0.71 to 0.65 forsegmentation. Drift synthesis results were compared to the
Common Corruptions Benchmark [31], showing more severe performance drops for common corruptions. Physicalfaithfulness of test cases greatly impacts model selection
decisions, with no overlap observed between top-3 training
data models for classification when comparing ISP and common corruptions (refer to numbers 1-3 in Figures 2 and 7).
Following ideas developed by Krikamol Muandet’s [57],data models could aid targeted generalization, serving task
models with the appropriate data model for each deployment
environment. Finally, drift synthesis permits validation without physical measurement, but requires access to raw data
and knowledge of the data model specification.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images
3 Drift adjustments with Φpara
Proc(a) Low intensity (0.001) X RAW with Φpara
Proc
LearnedValidation metricFrozen(b) High intensity (1.0) X RAW with Φpara
Proc
LearnedValidation metricRaw comparison

Drone

Microscopy

FrozenFigure 4: Low (a) and high (b) intensity images processed by a frozen and a learned pipeline. This type of drift optimization would not be possible with processed data. The
plots columns three and six display the mean of validation metrics over five cross validation runs. Column seven shows additional results on raw data for comparison. Error bars
are reported as one standard deviation. Optimization step 1439 and 915 correspond to epoch 60 into training.

2.2. Drift forensics
Products with machine learning components like medical
devices or autonomous vehicles require clear specificationof usage limitations. Our solution, Φpara
Proc , enables analysis
of task model susceptibility to dataset drift using adversarial
search, differentiating it from related work that applies gradient updates to individual images rather than data model
parameters. We identify risky data model parameter configurations for task model operation, reflecting potentialchanges in data model parameters like altering camera ISPs.
These parameters are kept within constraints of physicalfaithfulness while deteriorating task model performance.
Sensitivity of the classification task model to changes indata model parameters is depicted in Figure 3. Interestingly, larger changes in the resulting RGB images don’tnecessarily lead to the most severe task model performancedegradation. This emphasizes the need for precise data models for dataset drift validation. Practical use-cases of drift
forensics include providing a forensic signature to a licensee
detailing data model parameters that maintain task model
performance. However, performing drift forensics requires
access to raw data and data models.
2.3. Drift optimization
The previous section demonstrated how raw data and a differentiable data model can identify and test unfavorable data
models. We extend this concept to optimize the data itself,
creating a "beneficial drift". Two settings are considered:"learned", where both data and task model parameters are
optimized, and "frozen", where only task model parameters
are optimized. The "learned" model can increase accuracy
by up to 25% with less variance (Figure 4 (a)). However,this was not seen in low-resolution tasks like segmentation.
"Learned" models can also produce visual artifacts that may
improve stability and generalization. Similarly positive results were observed under varying conditions (Figure 4 (b)).
We’ve also investigated how parameterized data models
can optimize drift under constraints, and the potential oflearning directly on raw data for better task performance.
Optimizing drift can enhance task model performance andadapt traditionally human-optimized imaging pipelines for
ML models, beneficial in limited-resource situations. However, this doesn’t work for all tasks and requires raw sensor
access. The end goal may be to train on RAW data, withcurrent methods serving as transitional solutions.

3. Discussion
In this manuscript we studied the potential of differentiabledata models for images, paired with raw data, to control
dataset drift. This significant challenge affects numerous
machine learning disciplines. Drift synthesis enables thecreation of physically faithful drift test cases, leading toless severe performance drops. This allows model selection
and new ways to think about generalization. Drift synthesis could be beneficial for various domains including datasynthesis and precise data models [63; 62]. Drift forensics
identify and document data model limitations, enabling precision to satisfy regulatory constraints [27; 93; 59]. Drift optimization with differentiable data models enhances stability
and speed in learning, which could be valuable in areas suchas federated learning or domain adaptation [79; 78; 97; 10].
However, it may not work across all tasks. Lastly, raw data
usage needs to be more accessible to researchers for more
physically faithful data models [28; 64; 101; 99; 38; 74].
We release two raw image datasets to aid this endeavor. Better APIs to optical hardware can also help in providing moreaccessible raw data.
How far we can push the gradient into the real world is aninteresting future direction for data modelling. Including
more parts of the data acquisition hardware into the data
model and consequently the machine learning optimization
pipeline appears feasible [98] and represents an important
next step in aligning machine learning with real world data
infrastructures.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Use of Personal Data and Human Subjects The microscopy slides were purchased from a commercial lab vendor (J. Lieder GmbH & Co. KG, Ludwigsburg/Germany)
who attained consent. The drone dataset does not directly relate to people. Instances with potential PIIs such as faces or
license plates were removed. Full datasheet documentation
following [24] can be found in Appendix B.5.2.
Negative Societal Impact Machine learning risk management, such as the drift controls, can make ML deployment
possible and safer. More deployment translates to increases
in automation. A net risk-benefit analysis of automation
is beyond the scope of this manuscript. What we do know
is that steel can be cast into ploughs and swords. We areagainst the use of our findings for the latter purpose.

References[1] A. Abdelhamed, S. Lin, and M. S. Brown. A highquality denoising dataset for smartphone cameras. In
IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2018.
[2] B. Albertina, M. Watson, C. Holback, R. Jarosz,
S. Kirk, Y. Lee, and J. Lemmerman. Radiology datafrom the cancer genome atlas lung adenocarcinoma[tcga-luad] collection. The Cancer Imaging Archive,
2016.
[3] G. Aresta, T. Araújo, S. Kwok, S. S. Chennamsetty, M. Safwan, V. Alex, B. Marami, M. Prastawa,
M. Chan, M. Donovan, G. Fernandez, J. Zeineh,
M. Kohl, C. Walz, F. Ludwig, S. Braunewell,
M. Baust, Q. D. Vu, M. N. N. To, E. Kim, J. T. Kwak,
S. Galal, V. Sanchez-Freire, N. Brancati, M. Frucci,
D. Riccio, Y. Wang, L. Sun, K. Ma, J. Fang, I. Kone,
L. Boulmane, A. Campilho, C. Eloy, A. Polónia, and
P. Aguiar. Bach: Grand challenge on breast cancer
histology images. Medical Image Analysis, 56:122–
139, 2019.
[4] V. Ayyappan, A. Chang, C. Zhang, S. K. Paidi,
R. Bordett, T. Liang, I. Barman, and R. Pandey. Identification and staging of b-cell acute lymphoblastic
leukemia using quantitative phase imaging and machine learning. ACS Sensors, 5(10):3281–3289, 2020.
PMID: 33092347.
[5] A. Azulay and Y. Weiss. Why do deep convolutional
networks generalize so poorly to small image transformations? Journal of Machine Learning Research,
20:1–25, 2019.
[6] B. Bain. Diagnosis from the blood smear. The New
England journal of medicine, 353 5:498–507, 2005.

[7] B. E. Bayer. Color imaging array, July 20 1976. US
Patent 3,971,065.
[8] E. Beede, E. Baylor, F. Hersch, A. Iurchenko,
L. Wilcox, P. Ruamviboonsuk, and L. M. Vardoulakis.
A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic
retinopathy. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, page
1–12, New York, NY, USA, 2020. Association for
Computing Machinery.
[9] S. Beery, G. Van Horn, and P. Perona. Recognitionin terra incognita. In Proceedings of the European
Conference on Computer Vision (ECCV), pages 456–
473, 2018.
[10] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza,
F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine learning,
79(1):151–175, 2010.
[11] S. Biswas and S. Barma. A large-scale optical microscopy image dataset of potato tuber for deep learning based plant cell assessment. Scientific Data,
7(1):1–11, 2020.
[12] H. Bolhasani, E. Amjadi, M. Tabatabaeian, and S. J.
Jassbi. A histopathological image dataset for gradingbreast invasive ductal carcinomas. Informatics in
Medicine Unlocked, 19:100341, 2020.
[13] D. L. Bongiorno, M. Bryson, D. G. Dansereau, andS. B. Williams. Spectral characterization of COTS
RGB cameras using a linear variable edge filter. page
86600N, Burlingame, California, USA, Jan. 2013.
[14] T. A. Bubba, G. Kutyniok, M. Lassas, M. März,
W. Samek, S. Siltanen, and V. Srinivasan. Learning
the invisible: a hybrid deep learning-shearlet framework for limited angle computed tomography. Inverse
Problems, 35(6):064002, 2019.
[15] C. Chen, Q. Chen, J. Xu, and V. Koltun. Learning to see in the dark. 2018 IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages
3291–3300, 2018.
[16] P. Chlap, M. Hang, N. Vandenberg, J. Dowling,
L. Holloway, and A. Haworth. A review of medical image data augmentation techniques for deeplearning applications. Journal of Medical Imaging
and Radiation Oncology, 65, 06 2021.
[17] J. P. Cohen, M. Luck, and S. Honari. Distribution
matching losses can hallucinate features in medical
image translation. In Medical Image Computing
and Computer Assisted Intervention – MICCAI 2018,

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

pages 529–536. Springer International Publishing, 09
2018.
[18] R. Conrad and K. Narayan. Cem500k, a largescale heterogeneous unlabeled cellular electron microscopy image dataset for deep learning. eLife,10:e65894, apr 2021.
[19] D.-T. Dang-Nguyen, C. Pasquini, V. Conotter, and
G. Boato. Raise: A raw images dataset for digital
image forensics. In Proceedings of the 6th ACM
Multimedia Systems Conference, MMSys ’15, page219–224, New York, NY, USA, 2015. Association
for Computing Machinery.
[20] M. Dejean-Servières, K. Desnos, K. Abdelouahab,
W. Hamidouche, L. Morin, and M. Pelcat. Studyof the impact of standard image compression techniques on performance of image classification witha convolutional neural network. Research Reporthal-01725126, INSA Rennes; Univ Rennes; IETR;
Institut Pascal, 2017.
[21] S. Dodge and L. Karam. A study and comparison ofhuman and deep learning recognition performance
under visual distortions. In 2017 26th International
Conference on Computer Communication and Networks (ICCCN), pages 1–7, 2017.
[22] DroneDeploy.
Segmentation
Dataset.
https://github.com/dronedeploy/
dd-ml-segmentation-benchmark, 2019.
Accessed: 2021-09-19.
[23] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M.
Swetter, H. M. Blau, and S. Thrun. Dermatologistlevel classification of skin cancer with deep neuralnetworks. nature, 542(7639):115–118, 2017.
[24] T. Gebru, J. H. Morgenstern, B. Vecchione, J. W.
Vaughan, H. M. Wallach, H. Daumé, and K. Crawford. Datasheets for datasets. arXiv, 1803.09010,
2018.
[25] K. Goel, A. Gu, Y. Li, and C. Re. Model patching:
Closing the subgroup performance gap with data augmentation. In International Conference on Learning
Representations, 2021.
[26] B. Goyal, A. Dogra, S. Agrawal, B. Sohi, and
A. Sharma. Image denoising review: From classicalto state-of-the-art approaches. Information Fusion,
55:220–244, 2020.
[27] I. S. W. Group et al. Software as a medical device(samd): Application of quality management system,
2018.

[28] HAMAMATSU. ORCA-Flash4.0 V3 Digital CMOS
camera C13440-20CU - Technical note. HAMAMATSU.
[29] S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T.
Barron, F. Kainz, J. Chen, and M. Levoy. Burst
photography for high dynamic range and low-light
imaging on mobile cameras. ACM Transactions on
Graphics (Proc. SIGGRAPH Asia), 35(6), 2016.
[30] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual
learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 770–778, 2016.
[31] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions andperturbations. Proceedings of the International Conference on Learning Representations, 2019.
[32] D. Hendrycks and T. Dietterich. Benchmarking
neural network robustness to common corruptions
and perturbations. arXiv preprint arXiv:1903.12261,
2019.
[33] L. Hou, R. Gupta, J. S. Van Arnam, Y. Zhang,
K. Sivalenka, D. Samaras, T. M. Kurc, and J. H.
Saltz. Dataset of segmented nuclei in hematoxylin
and eosin stained histopathology images of ten cancer
types. Scientific data, 7(1):1–12, 2020.
[34] R. W. G. Hunt and M. R. Pointer. Measuring colour.
John Wiley & Sons, 2011.
[35] R. Jaroensri, C. Biscarrat, M. Aittala, and F. Durand. Generating training data for denoising real
rgb images via camera pipeline simulation. arXiv,
1904.08825, 2019.
[36] X. Jia, Y. Cao, D. O’Connor, J. Zhu, D. C. Tsang,
B. Zou, and D. Hou. Mapping soil pollution by using
drone image recognition and machine learning at anarsenic-contaminated agricultural field. Environmental Pollution, 270:116281, 2021.
[37] Y.-Y. Jo, Y. S. Choi, H. W. Park, J. H. Lee, H. Jung,
H.-E. Kim, K. Ko, C. W. Lee, H. S. Cha, andY. Hwangbo. Impact of image compression on deeplearning-based mammogram classification. Scientific
Reports, 11(1):1–9, 2021.
[38] A. Karpathy. Tesla ai day 2021.
[39] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, 1412.6980, 2015.
[40] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie,
M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga,

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

R. L. Phillips, I. Gao, T. Lee, E. David, I. Stavness,
W. Guo, B. Earnshaw, I. Haque, S. M. Beery,J. Leskovec, A. Kundaje, E. Pierson, S. Levine,
C. Finn, and P. Liang. Wilds: A benchmark of in-thewild distribution shifts. In M. Meila and T. Zhang,
editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5637–
5664. PMLR, 18–24 Jul 2021.
[41] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie,
M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga,
R. L. Phillips, I. Gao, T. Lee, E. David, I. Stavness,
W. Guo, B. Earnshaw, I. Haque, S. M. Beery,J. Leskovec, A. Kundaje, E. Pierson, S. Levine,
C. Finn, and P. Liang. Wilds: A benchmark of inthe-wild distribution shifts. In Proceedings of the
38th International Conference on Machine Learning,volume 139 of Proceedings of Machine Learning
Research, pages 5637–5664. PMLR, 2021.
[42] M. Kulbacki, J. Segen, W. Knieć, R. Klempous,
K. Kluwak, J. Nikodem, J. Kulbacka, and A. Serester. Survey of drones for agriculture automationfrom planting to harvest. In 2018 IEEE 22nd International Conference on Intelligent Engineering Systems(INES), pages 000353–000358. IEEE, 2018.
[43] R. D. Labati, V. Piuri, and F. Scotti. All-idb: The
acute lymphoblastic leukemia image database forimage processing. In 2011 18th IEEE International
Conference on Image Processing, pages 2045–2048,
2011.
[44] X. Li, B. Gunturk, and L. Zhang. Image demosaicing:A systematic survey. In Visual Communications andImage Processing 2008, volume 6822, page 68221J.
International Society for Optics and Photonics, 2008.
[45] W. Liang and J. Zou. Metashift: A dataset of datasetsfor evaluating contextual distribution shifts and training conflicts. In International Conference on Learning Representations, 2022.
[46] Library of Congress.
Camera Raw Formats(Group Description). https://www.loc.gov/
preservation/digital/formats/fdd/
fdd000241.shtml, Dec. 2016.
Accessed:
2020-11-03.
[47] S. Longanbach, M. Miers, E. Keohane, L. Smith, andJ. Walenga. Rodak’s hematology: Clinical principles
and applications. 2016.
[48] Y. Lyu, G. Vosselman, G.-S. Xia, A. Yilmaz, and
M. Y. Yang. Uavid: A semantic segmentation dataset
for uav imagery. ISPRS Journal of Photogrammetry
and Remote Sensing, 165:108–119, 2020.

[49] A. Maier, H. Köstler, M. Heisig, P. Krauss, and S. H.
Yang. Known operator learning and hybrid machinelearning in medical imaging—a review of the past,
the present, and the future. Progress in Biomedical
Engineering, 2022.
[50] A. Maier, H. Köstler, M. Heisig, P. Krauss, and S. H.
Yang. Known operator learning and hybrid machinelearning in medical imaging — a review of the past,
the present, and the future. arXiv, 2108.04543, 2021.
[51] A. K. Maier, C. Syben, B. Stimpel, T. Würfl, M. Hoffmann, F. Schebesch, W. Fu, L. Mill, L. Kling, and
S. Christiansen. Learning with known operators reduces maximum error bounds. Nature machine intelligence, 1(8):373–380, 2019.
[52] M. Maimaitijiang, V. Sagan, P. Sidike, S. Hartling,
F. Esposito, and F. B. Fritschi. Soybean yield prediction from uav using multimodal data fusion anddeep learning. Remote Sensing of Environment,
237:111599, 2020.
[53] A. Marcu, D. Costea, V. Licaret, and M. Leordeanu.
Towards automatic annotation for semantic segmentation in drone videos. arXiv, 1910.10026, 2019.
[54] C. Matek and C. Marr. Robustness evaluation of a
convolutional neural network for the classification of
single cells in acute myeloid leukemia. In ICLR 2021,
RobustML workshop, 2020.
[55] C. Matek, S. Schwarz, C. Marr, and K. Spiekermann.
A single-cell morphological dataset of leukocytes
from aml patients and non-malignant controls (amlcytomorphology_lmu). The Cancer Imaging Archive(TCIA), 2019.
[56] C. Matek, S. Schwarz, K. Spiekermann, and C. Marr.
Human-level recognition of blast cells in acute
myeloid leukaemia with convolutional neural networks. Nature Machine Intelligence, 1(11):538–544,
2019.
[57] K. Muandet. Impossibility of collective intelligence,2022.
[58] R. Nguyen, D. K. Prasad, and M. S. Brown. Raw-toraw: Mapping between image sensor color responses.
In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pages 3398–3405,
2014.
[59] U. NSTC. Ensuring american leadership in automated vehicle technologies: Automated vehicles 4.0.
Las Vegas. Recuperado el, 25:2020–02, 2020.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images[60] L. Oala, M. Aversa, G. Nobis, K. Willis, Y. Neuenschwander, M. Buck, C. Matek, J. Extermann, E. Pomarico, W. Samek, R. Murray-Smith, C. Clausen,
and B. Sanguinetti. Data models for dataset drift
controls in machine learning with optical images.
Transactions on Machine Learning Research, 2023.
[61] L. Oala, J. Fehr, L. Gilli, P. Balachandran, A. W.
Leite, S. Calderon-Ramirez, D. X. Li, G. Nobis,
E. A. M. Alvarado, G. Jaramillo-Gutierrez, et al.
Ml4h auditing: From paper to practice. In Machinelearning for health, pages 280–317. PMLR, 2020.
[62] L. Oala, C. Heiß, J. Macdonald, M. März, G. Kutyniok, and W. Samek. Detecting failure modes inimage reconstructions with interval neural network
uncertainty. International Journal of Computer Assisted Radiology and Surgery, 16(12):2089–2097,
2021.
[63] A. Oliver, A. Odena, C. Raffel, E. Cubuk, and
I. Goodfellow.
Realistic evaluation of semisupervised learning algortihms. In International conference on Learning Representations, pages 1–15,
2018.
[64] PerkinElmer. TotalChrom Workstation User’s Guide- Volume I. PerkinElmer.
[65] B. Phan, F. Mannan, and F. Heide. Adversarial imaging pipelines. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pages 16051–16061, 2021.
[66] E. Pomarico, C. Schmidt, F. Chays, D. Nguyen,
A. Planchette, A. Tissot, A. Roux, L. Batti,
C. Clausen, T. Lasser, et al. Statistical distortion ofsupervised learning predictions in optical microscopy
induced by image compression. Scientific reports,
12(1):1–10, 2022.
[67] M. Poyser, A. Atapour-Abarghouei, and T. P.
Breckon. On the impact of lossy image and video
compression on the performance of deep convolutional neural network architectures. In 2020 25th
International Conference on Pattern Recognition(ICPR), pages 2830–2837. IEEE, 2021.
[68] J. Ragan-Kelley, C. Barnes, A. Adams, S. Paris,
F. Durand, and S. Amarasinghe. Halide: a language
and compiler for optimizing parallelism, locality, andrecomputation in image processing pipelines. Acm
Sigplan Notices, 48(6):519–530, 2013.
[69] S. Ratnasingam. Deep camera: A fully convolutional
neural network for image signal processing. In 2019
IEEE/CVF International Conference on Computer

Vision Workshop (ICCVW), pages 3868–3878, Los
Alamitos, CA, USA, oct 2019. IEEE Computer Society.
[70] E. Riba, D. Mishkin, D. Ponsa, E. Rublee, and
G. Bradski. Kornia: an open source differentiable
computer vision library for pytorch. In Winter Conference on Applications of Computer Vision, 2020.
[71] M. Ronchetti. Torchradon: Fast differentiable routines for computed tomography. arXiv, 2009.14788,
2020.
[72] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention,
pages 234–241. Springer, 2015.
[73] A. Rowlands. Physics of digital photography. IOP
Publishing, 2017.
[74] G. Rudolph and U. Voelzke. Three sensor types driveautonomous vehicles, 2017.
[75] L. Ruff, J. R. Kauffmann, R. A. Vandermeulen,
G. Montavon, W. Samek, M. Kloft, T. G. Dietterich,and K.-R. Müller. A unifying review of deep andshallow anomaly detection. Proceedings of the IEEE,
2021.
[76] O. Russakovsky, J. Deng, H. Su, J. Krause,
S. Satheesh, S. Ma, Z. Huang, A. Karpathy,
A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei.
Imagenet large scale visual recognition challenge. Int.
J. Comput. Vision, 115(3):211–252, dec 2015.
[77] N. Sambasivan, S. Kapania, H. Highfill, D. Akrong,
P. Paritosh, and L. M. Aroyo. “everyone wants todo the model work, not the data work”: Data cascades in high-stakes ai. In Proceedings of the 2021
CHI Conference on Human Factors in Computing
Systems, New York, NY, USA, 2021. Association for
Computing Machinery.
[78] F. Sattler, K.-R. Müller, T. Wiegand, and W. Samek.
On the byzantine robustness of clustered federated
learning. In ICASSP 2020-2020 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8861–8865. IEEE, 2020.
[79] F. Sattler, S. Wiedemann, K.-R. Müller, and
W. Samek. Robust and communication-efficient federated learning from non-iid data. IEEE transactions
on neural networks and learning systems, 31(9):3400–
3413, 2019.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images[80] F. Schiffers, Z. Yu, S. Arguin, A. Maier, and Q. Ren.
Synthetic fundus fluorescein angiography using deepneural networks. In A. Maier, T. M. Deserno, H. Handels, K. H. Maier-Hein, C. Palm, and T. Tolxdorff,
editors, Bildverarbeitung für die Medizin 2018, pages
234–238, Berlin, Heidelberg, 2018. Springer Berlin
Heidelberg.
[81] F. Schill. pyraw. https://github.com/
fschill/pyraw, 2015.
[82] A. Shamsoshoara, F. Afghah, A. Razi, L. Zheng,
P. Z. Fulé, and E. Blasch. Aerial imagery pile burndetection using deep learning: The flame dataset.
Computer Networks, 193:108001, 2021.

[91] D. Tellez, G. Litjens, P. Bándi, W. Bulten, J.-M.
Bokhorst, F. Ciompi, and J. van der Laak. Quantifying the effects of data augmentation and stain color
normalization in convolutional neural networks forcomputational pathology. Medical Image Analysis,
58:101544, 2019.
[92] C. Tian, L. Fei, W. Zheng, Y. Xu, W. Zuo, and C.-W.
Lin. Deep learning on image denoising: An overview.
Neural Networks, 131:251–275, 2020.
[93] A. TIR57. Principles for medical device security—risk management. Arlington, VA: Association
for the Advancement of Medical Instrumentation,
2016.

[83] A. Subbaswamy, R. Adams, and S. Saria. Evaluating
model robustness and stability to dataset shift. In
A. Banerjee and K. Fukumizu, editors, Proceedingsof The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedingsof Machine Learning Research, pages 2611–2619.
PMLR, 13–15 Apr 2021.

[94] E. Tseng, A. Mosleh, F. Mannan, K. St-Arnaud,
A. Sharma, Y. Peng, A. Braun, D. Nowrouzezahrai,
J.-F. Lalonde, and F. Heide. Differentiable compound
optics and processing pipeline optimization for endto-end camera design. ACM Trans. Graph., 40(2),June 2021.

[84] A. Subbaswamy, R. Adams, and S. Saria. Evaluating
model robustness and stability to dataset shift. In
International Conference on Artificial Intelligenceand Statistics, pages 2611–2619. PMLR, 2021.

[95] TU Graz.
ICG - DroneDataset.
https://www.tugraz.at/index.php?id=22387,
2019. Accessed: 2021-05-06.

[85] S. M. Swetter. Artificial intelligence may improvemelanoma detection. Dermatology Times, 41(9):36,
2020.
[86] C. Syben, M. Michen, B. Stimpel, S. Seitz, S. Ploner,and A. K. Maier. Pyro-nn: Python reconstruction operators in neural networks. Medical physics,
46(11):5110–5115, 2019.
[87] N.-S. Syu, Y.-S. Chen, and Y.-Y. Chuang. Learning
deep convolutional networks for demosaicing. arXiv,
1802.03769, 2018.
[88] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna,
D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint
arXiv:1312.6199, 2013.
[89] S. Tani, Y. Fukunaga, S. Shimizu, M. Fukunishi,
K. Ishii, and K. Tamiya. Color Standardization
Method and System for Whole Slide Imaging Basedon Spectral Sensing. Analytical Cellular Pathology,
35(2):107–115, 2012.
[90] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht,
and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. In Advancesin Neural Information Processing Systems (NeurIPS),
2020.

[96] G. Verhoeven. It’s all about the format–unleashing
the power of raw aerial photography. International
Journal of Remote Sensing, 31(8):2009–2042, 2010.
[97] H. Wang, Z. Kaplan, D. Niu, and B. Li. Optimizing
federated learning on non-iid data with reinforcement learning. In IEEE INFOCOM 2020 - IEEE
Conference on Computer Communications, pages
1698–1707, 2020.
[98] L. G. Wright, T. Onodera, M. M. Stein, T. Wang, D. T.
Schachter, Z. Hu, and P. L. McMahon. Deep physical neural networks trained with backpropagation.
Nature, 601(7894):549–555, 2022.
[99] D. J. Yeong, G. Velasco-Hernandez, J. Barry, andJ. Walsh. Sensor and sensor fusion technology in autonomous vehicles: A review. Sensors, 21(6), 2021.
[100] F. G. Zanjani, S. Zinger, B. Piepers, S. Mahmoudpour, P. Schelkens, and P. H. N. de With. Impact
of JPEG 2000 compression on deep convolutional
neural networks for metastatic cancer detection inhistopathological images. Journal of Medical Imaging, 6(2):1 – 9, 2019.
[101] ZEISS. Exporting Images and Movies in ZEN Blue.
ZEISS.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images[102] B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen, and
M. S. Rosen. Image reconstruction by domaintransform manifold learning. Nature, 555(7697):487–
492, 2018.
[103] P. Zhu, L. Wen, D. Du, X. Bian, H. Fan, Q. Hu,
and H. Ling. Detection and tracking meet drones
challenge. IEEE Transactions on Pattern Analysis
and Machine Intelligence, (01):1–1, Oct. 2021.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

A. AppendicesB. Preliminaries: a data model for images
Before proceeding with a description of the methods we use to obtain the data models ΦProc in this study, let us briefly
review the distinction between raw data xRAW , processed image v and the mechanisms ΦProc ∶ RH,W → RC,H,W by which
image data transitions between these states3 . Image acquisition has traditionally been optimized for the human perceptionof a scene [34; 73]. Human eyes detect only the visible spectrum of electromagnetic radiation, hence imaging cameras
in different application domains such as medical imaging or remote sensing are usually calibrated to aid the human eye
perform a downstream task. This process that gives rise to optical image data, which ultimately forms the backbone for any
machine learning downstream, is rarely considered in the machine learning literature. Conversely, most research to date has
been conducted on processed RGB image representations. The raw sensor image xRAW obtained from a camera differs
substantially from the processed image that is used in conventional machine learning pipelines. The xRAW state appears
like a grey scale image with a grid structure (see xraw in Figure 1). This grid is given by the Bayer color filter mosaic,
which lies over sensors [7]. The final RGB image v is the result of a series of transformations applied to xRAW . For many
steps in this process different possible algorithms exist. Starting from a single xRAW , all those possible combinations cangenerate an exponential number of possible images that are slightly different in terms of colors, lighting and blur - variations
that contribute to dataset drift. In Figure 1 a conventional pipeline from xRAW to the final RGB image v is depicted. Here,common and core transformations are considered. Note that depending on the application context it is possible to reorder or
add additional steps. The symbol Φi is used to denote the ith transformation and v i (view) for the output image of Φi . The
first step of the pipeline is black level correction ΦBL , which removes any constant offset. The image v BL is a grey image
with a Bayer filter pattern. A demosaicing algorithm ΦDM is applied to construct the full RGB color image [44]. Givenv DM , intensities are adjusted to obtain a neutrally illuminated image v WB through a white balance transformation ΦWB .
By considering color dependencies, a color correction transformation ΦCC is applied to balance hue and saturation of theimage. Once lighting and colors are corrected, a sharpening algorithm ΦSH is applied to reduce image blurriness. Thistransformation can make the image appear more noisy. For this reason a denoising algorithm ΦDN is applied afterwards[26; 92]. Finally, gamma correction, ΦGC , adjusts the linearity of the pixel values. For a closed form description of thesetransformations see Section 1.2. Compression may also take place as an additional step. It is not considered here as the
input image size is already small. Furthermore, the effect of compression on downstream task model performance has beenthoroughly examined before [20; 37; 100; 67; 66]. However, users of our code can add this step or reorder the sequence ofsteps in the modular processing object class per their use case needs4 .
B.1. Data models details
The second ingredient to this study are the data models of image processing. Image data transitions from a raw state xRAW toprocessed image v via image signal processing ΦProc ∶ RH,W → RC,H,W [73]. Raw sensor images (xRAW ) from cameras aredifferent from conventional machine learning input, appearing as a grey scale image with a grid structure due to the Bayer
color filter mosaic [7]. These images undergo multiple transformations to form the final RGB image v, generating numerous
slightly different images, contributing to dataset drift. The process is illustrated in Figure 1, including transformations
such as black level correction, demosaicing, white balance, color correction, sharpening, denoising, and gamma correction[44; 26; 92]. Compression, not discussed here, is also applicable and can be added or reordered in the modular processing
object class as per use-case requirements.
Let (X RAW , Y ) ∶ Ω → RH,W × Y be the raw sensor data generating random variable on some probability space (Ω, F, P),
with Y = {0, 1}K for classification and Y = {0, 1}H,W for segmentation. Let ΦTask ∶ RC,H,W → Y be the task model
determined during training. The inputs that are given to the task model ΦTask are the outputs of the data model ΦProc . We
distinguish between the raw sensor image xRAW and a view v = ΦProc (xRAW ) of this image, where ΦProc ∶ RH,W → RC,H,W
models the transformation steps applied to the raw sensor image during processing.
The objective in supervised machine learning is to learn a task model ΦTask ∶ RC,H,W → Y within a fixed class of task
models H that minimizes the expected loss wrt. the loss function L ∶ Y × Y → [0, ∞), that is to find Φ⋆Task such thatinf E[L (ΦTask (V ), Y )]

ΦTask ∈H
3
4

We recommend [73] for a good introduction to the physics of digital optical imaging.
See pipeline_torch.py and pipeline_numpy.py in our code.

(8)

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Data models

Used functions

bi,s,mebi,s,gabi,u,mebi,u,game,s,meme,s,game,u,meme,u,gama,s,mema,s,gama,u,mema,u,ga

ΦBil
DM
ΦBil
DM
ΦBil
DM
ΦBil
DM
ΦMen
DM
ΦMen
DM
ΦMen
DM
ΦMen
DM
ΦMal
DM
ΦMal
DM
ΦMal
DM
ΦMal
DM

ΦSF
SH
ΦSF
SH
ΦUM
SH
ΦUM
SH
ΦSF
SH
ΦSF
SH
ΦUM
SH
ΦUM
SH
ΦSF
SH
ΦSF
SH
ΦUM
SH
ΦUM
SH

ΦMD
DN
ΦGD
DN
ΦMD
DN
ΦGD
DN
ΦMD
DN
ΦGD
DN
ΦMD
DN
ΦGD
DN
ΦMD
DN
ΦGD
DN
ΦMD
DN
ΦGD
DN

Table 1: Abbreviations of the twelve configurations of the static data model Φstat
Proc used in the drift synthesis experiments.
is attained. Towards that goal, ΦTask is determined during training such that the empirical error
1 N∑ L (ΦTask (v n ), yn )
N n=1

(9)

is minimized over a sample S = ((v 1 , y1 ), ..., (v N , yN )) of views. Modelling in the conventional machine learning
setting begins with the image data generating random variable (V , Y ) = (ΦProc (X RAW ), Y ) and the target distribution
Dt = P ○ (V , Y )−1 . Given a dataset drift Ds = P ○ (Ṽ , Y )−1 =/ Dt , as specified in Equation (1), without a data model wehave little recourse to disentangle reasons for performance drops in ΦTask . To alleviate this underspecification, an explicit
para
data model is needed. We consider two such models in this study: a static model Φstat
Proc and a parametrized model ΦProc .

In the following, we denote by xRAW ∈ [0, 1]H,W the normalized raw image, that is a grey scale image with a Bayer filter
pattern normalized by 216 − 1, i.e.
⎡ A1,1⎢⎢⎢ .
⎢xRAW = ⎢
⎢ .
⎢ .
⎢
⎢
⎢A H ,1
⎣
2

.
.

.

.

.
.

.

.
.

A1, W ⎤
⎥
2 ⎥
. ⎥
⎥
. ⎥
⎥
. ⎥
⎥
⎥
AH ,W ⎥
2
2 ⎦rwith Ah,j = [ 2h+1,2w+1
g2h,2w+1

g2h+1,2w],b2h,2w(10)

where the values r2h+1,2w+1 , g2h+1,2w , g2h,2w+1 , b2h,2w correspond to the values measured through the different sensors andnormalized by 216 − 1. We provide here a precise description of the transformations that we consider in our static model
para
Φstat
Proc , followed by a description how to convert this static model into a differentiable, parametrized model ΦProc .
B.1.1. T HE STATIC DATA MODEL ΦSTAT
P ROC
Following common steps in ISP, the static data model is defined as the compositionΦstat
Proc = ΦGC ○ ΦDN ○ ΦSH ○ ΦCC ○ ΦWB ○ ΦDM ○ ΦBL ,(11)

mapping a raw sensor image to a RGB image. We note that other data model variations, for example by reordering or adding
steps, are feasible. The static data models allow the controlled synthesis of different, physically faithful views from thesame underlying raw sensor data by manually changing the configurations of the intermediate steps. Fixing the continuousfeatures, but varying ΦDM , ΦSH and ΦDN results in twelve different views for the configurations considered here. Samples
for each of the twelve data models are provided in Figure 5. The individual functions of the composition Φstat
Proc can be foundin Appendix B.1.3.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Figure 5: Samples for both datasets, Raw-Microscopy and Raw-Drone, from all twelve static data models Φstat
Proc used forthe drift synthesis experiments in Section 2.1. A version with higher resolution is omitted here to save space and can instead
be found in Figure 6 in the appendices.
An overview of the data model configurations and their corresponding abbreviations can be found alongside processed
samples in Table 1 and Figure 5.
B.1.2. T HE PARAMETRIZED DATA MODEL ΦPARA
P ROC
For a fixed raw sensor image, the parametrized data model Φpara
Proc maps from a parameter space Θ to a RGB image. It issimilar to the static data model with the notable difference that each processing step is differentiable wrt. its parameters θ.
This allows for backpropagation of the gradient from the output of the task model ΦTask through the data model ΦProc all theway back to the raw sensor image xRAW to perform drift forensics and drift adjustments. Hence, we aim to design a data
H,W
model Φpara× Θ → RC,H,W that is differentiable in θ ∈ Θ satisfying
Proc ∶ R
para
stat
)
Φstat
Proc = ΦProc (⋅, θ(12)for some choice of parameters θ stat and some fixed configuration of the static pipeline Φstat
Proc . Using the individual functional
components specified in Appendix B.1.4, we define for θ = (θ 1 , ..., θ 7 ) ∈ Θ the parametrized processing model
3,H,W
Φpara
× Θ → [0, 1]3,H,W , (xRAW , θ) ↦ v
P roc ∶ [0, 1](13)

by the composition
para

para

para

para

para

para

para

v = (ΦGC (⋅, θ7 ) ○ ΦDN (⋅, θ6 ) ○ ΦSH (⋅, θ5 ) ○ ΦCC (⋅, θ4 ) ○ ΦWB (⋅, θ3 ) ○ ΦDM (⋅, θ2 ) ○ ΦBL (⋅, θ1 )) (xRAW ) .

(14)

The operations used above are differentiable except for the clipping operation in the GC that is a.e.-differentiable5 , since theset {0, 1} of non-differentiable points has measure zero. Assuming in addition that P ((vDN )c,h,w ∈ {0, 1}) = 0 holds true
for the entries of v DN results in an a.e.-differentiable processing model. We further say that Φpara
Proc is differentiable, notingthat this holds only a.e. under the aforementioned assumption.
B.1.3. S TATIC DATA MODEL ΦSTAT
P ROC
If not stated otherwise, writing the equation vc,h,w = ac,h,w + bc,h,w defines vc,h,w for all 1 ≤ c ≤ 3, 1 ≤ h ≤ H and1 ≤ h ≤ W.
Black level correction (BL) removes thermal noise and readout noise generated from the camera sensor. The transformation
is given byΦBL ∶ [0, 1]H,W → [0, 1]H,W , xRAW ↦ v BL ,(15)
5a.e. stands for almost everywhereData Models for Dataset Drift Controls in Machine Learning With Optical Images

with(vBL )2h+1,2w+1 = x2h+1,2w+1 − bl1(vBL )2h,2w+1 = x2h,2w+1 − bl2(vBL )2h+1,2w = x2h+1,2w − bl3(vBL )2h,2w = x2h,2w − bl4 ,By design of bl ∈ R4 , black level correction ensures that v BL is again an element of [0, 1]H,W .
Demosaicing (DM) is applied to reconstruct the full RGB color image through interpolation. We use one out of theMenMalthree demosaicing algorithms BayerBilinear (ΦBil
DM ), Menon2007 (ΦDM ) and Malvar2004 (ΦDM ) from the Python packagecolor-demosaicing and denote this transformation by the mapΦDM ∶ [0, 1]H,W → [0, 1]3,H,W , v ↦ v DM .

(16)

White balance (WB) is applied to obtain a neutrally illuminated image. The transformation is given byΦWB ∶ [0, 1]3,H,W → [0, 1]3,H,W , v ↦ v WB ,(17)

(vWB )c,h,w = wbc ⋅ (vDM )c,h,w .

(18)

where wb ∈ [0, 1]3 adjusts the intensities byColor correction (CC) balances the saturation of the image by considering color dependencies. Let M ∈ R3,3 be the color
matrix. The transformation is defined bywhereΦCC ∶ [0, 1]3,H,W → R3,H,W , v ↦ v CC ,

(19)

⎤
⎡(v )
⎤
⎡(v )
⎢ WB 1,h,w ⎥⎢ CC 1,h,w ⎥⎥⎢⎥⎢v CC = ⎢(vCC )2,h,w ⎥ = M ⎢(vWB )2,h,w ⎥ .
⎥⎢⎥⎢⎢(vWB )3,h,w ⎥⎢(vCC )3,h,w ⎥
⎦
⎣
⎦
⎣

(20)

The entries of the resulting v CC are no longer restricted to [0, 1].

Sharpening (SH) reduces the blurriness of an image. We use the two methods sharpening filter (ΦSF
SH ) and unsharp
masking (ΦUM
)
that
are
appliedafter
atransformation
of
the
view
v
to
the
Y
U
V
-colorspace.
To
convertthe view to theCCSHY U V -color space we use the skimage.color function rgb2yuv (ΦY U V ). The sharpening filter
SF ∶ R3,H,W → R3,H,W ,(21)

is defined by a channel-wise convolution(SF (v))c,h,w = ((v c ⋆ k)h,w )cof the view⎡0⎢⎢with k ∶= ⎢−1
⎢
⎢0
⎣

−1
5
−1

0⎤
⎥
⎥
−1⎥
⎥
0⎥
⎦

v = ΦY U V (v CC ).

(22)

(23)

For unsharp masking we use the ski.filters function unsharp_mask modeled by U M . To formally define the
sharpening we writeΦSH ∶ R3,H,W → R3,H,W , v ↦ v SH
(24)
where

v SH = algo ○ ΦY U V (v CC ) with algo ∈ {SH, U M }.

(25)

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Denoising (DN) reduces the noise in an image that is (partly) introduced by SH and transforms the Y U V -color space view
back to the RGB-color space. For the latter transformation, the skimage.color function yuv2rgb (Φ−1
Y U V ) is used.
GD
We apply one out of the two methods Gaussian denoising (ΦGD
DN ) and Median denoising (ΦDN ). For Gaussian denoising, weapply a Gaussian filter (GF) with standard deviation of σ = 0.5 from the scipy.ndimage package. For median denoising
we apply a median filter (MF of size 3 from the scipy.ndimage package. Formally, this reads as

whereΦDN ∶ R3,H,W → R3,H,W , v ↦ v DN

(26)

v DN = Φ−1
Y U V ○ algo(v SH ) with algo ∈ {GF, U M }.

(27)

Gamma correction (GC) equilibrates the overall brightness of the image. First, the entries of the view v DN are clipped to[0, 1] leading to(vCP )c,h,w = (vDN )c,h,w 1{0≤(vDN )c,h,w ≤1} + 1{(vDN )c,h,w >1} .
(28)
Second, the brightness adjusting transformation is defined byΦGC ∶ R3,H,W → [0, 1]3,H,W , v ↦ v GC = (v CP ) γ

1

(29)

for some γ > 0 applied element-wise. Note that zero-clipping is necessary for v GC to be well-defined.
In total, we define the composition

H,W
Φstat
↦ [0, 1]3,H,W
P roc ∶ [0, 1]

(30)

Φstat
Proc ∶= ΦGC ○ ΦDN ○ ΦSH ○ ΦCC ○ ΦWB ○ ΦDM ○ ΦBL(31)of the above steps
and call Φstat
Proc the static pipeline.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Figure 6: Samples for both datasets, Raw-Microscopy and Raw-Drone, from all twelve pipelines used in the drift synthesis
experiments. The legend for abbreviations can be found in Table 1.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

B.1.4. PARAMETRIZED DATA MODEL ΦPARA
P ROC
Black level correction (BL) For the parametrized black level correction define the map
H,W
Φstat
× R4 → RH,W , (xRAW , θ 1 ) ↦ v BL = ΦBL (xRAW )∣bl=θ1 .
BL ∶ [0, 1](32)

and set Θ1 ∶= R4 .

Demosaicing (DM) We first convert v BL to a three channel image [R, G, B] ∈ R3,H,W where the entries of R, G and B
are zero except
R2h+1,2w+1 = v BL2h+1,2w+1 ,B2h,2w = vBL2h,2w ,G2h+1,2w = vBL2h+1,2w ,G2h,2w+1 = vBL2h,2w+1 .

To parametrize ΦBil
DM define the map
H,W
Φpara
× R3,3,3 → R3,H,W , (v BL , θ 2 ) ↦ v DM
DM ∶ [0, 1](33)

with θ 2 = [k1 , k2 , k3 ], where the kernels k1 , k2 , k3 ∈ R3,3 are separately applied to each color channel resuling inv DM1,h,w = (R ⋆ k1 )h,w

v DM2,h,w = (G ⋆ k2 )h,w

v DM3,h,w = (B ⋆ k3 )h,w .

The source code of BayerBilinear shows that the parameter choice⎡ 0⎢⎢
k1 = k3 = ⎢0.25⎢⎢ 0⎣leads to

0.25
1
0.25

0 ⎤
⎥
⎥
0.25⎥
⎥
0 ⎥
⎦

and

⎡0.25
⎢
⎢
k2 = ⎢ 0.5
⎢
⎢0.25
⎣

0.5 0.25⎤
⎥
⎥
1
0.5 ⎥
⎥
0.5 0.25⎥
⎦

para
ΦBil
DM = ΦDM (⋅, θ 2 ).

(34)

(35)

Towards the definition of the parameter space set Θ2 ∶= R3,3,3 × Θ1 .

White balance (WB) For the parametrized white balance define the map
3,H,W
Φpara
× R3 → R3,H,W , (v DM , θ 3 ) ↦ v WB = ΦWB (v DM )∣wb=θ3
WB ∶ R(36)

and set Θ3 ∶= R3 × Θ2 .
Color correction (CC) For the parametrized color correction define the map
3,H,W
Φpara
× R3,3 → R3,H,W , (v WB , θ 4 ) ↦ v CC = ΦCC (vWB )∣M =θ4
CC ∶ R(37)

and set Θ4 ∶= R3,3 × Θ3
Sharpening (SH) We parametrize the sharpening filter configuration of the static pipeline, by using the entries of k ∈ R3,3
defined in (22) as parameters leading to
3,H,W
Φpara
× R3,3 → R3,H,W , (v CC , θ 5 ) ↦ v SH = ΦSH (vCC )∣k=θ5
SH ∶ R(38)

and Θ5 ∶= R3,3 × θ4 .
Denoising (DN) We parametrize the configuration where the Gaussian denoising method is applied. Applying the Gaussian
filter from scipy.ndimage with σ = 0.5 is equivalent to a convolution of the view in the Y U V -color space with a
specific kgauss ∈ R5,5 . For the specific values of kgauss see K_BLUR at the code of the parametrized pipeline. Therefore, toparametrize DN we define the map
3,H,W
Φpara
× R5,5 → R3,H,W , (v SH , θ 6 ) ↦ v DN = ΦDN (vSH )∣kgauss =θ6
DN ∶ R(39)

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

and set Θ6 ∶= R5,5 × Θ5
Gamma correction (GC) Define the parametrized gamma correction by
3,H,W
Φpara
× R → [0, 1]3,H,W , (v DN , θ7 ) ↦ v = v GC = ΦGC (v DN )∣γ=θ7 .
GC ∶ R(40)

The following values were used to initialize Φpara
Proc (both "Frozen" and "Learned") in experiment Section 2.3:
12class ParametrizedProcessing(nn.Module):"""Differentiable processing pipeline via torch transformations

3
4
5
6
7
8

Args:
camera_parameters (tuple(list), optional): applies given camera parameters inprocessing
track_stages (bool, optional): whether or not to retain intermediary steps inprocessing
batch_norm_output (bool, optional): adds a BatchNorm layer to the end of theprocessing
"""

9
10
11
12
13
14

def __init__(self, camera_parameters=None, track_stages=False, batch_norm_output=True):super().__init__()self.stages = Noneself.buffer = Noneself.track_stages = track_stages

15

if camera_parameters is None:camera_parameters = DEFAULT_CAMERA_PARAMS

16
17
18

black_level, white_balance, colour_matrix = camera_parameters

19
20

self.black_level = nn.Parameter(torch.as_tensor(black_level))self.white_balance = nn.Parameter(torch.as_tensor(white_balance).reshape(1, 3))self.colour_correction = nn.Parameter(torch.as_tensor(colour_matrix).reshape(3, 3)

21
22
23

)
24

self.gamma_correct = nn.Parameter(torch.Tensor([2.2]))

25
26

self.debayer = Debayer()27
28

self.sharpening_filter = nn.Conv2d(1, 1, kernel_size=3, padding=1, bias=False)
self.sharpening_filter.weight.data[0][0] = K_SHARP.clone()

29
30
31
32
33

self.gaussian_blur = nn.Conv2d(1, 1, kernel_size=5, padding=2, padding_mode=’reflect’, bias=False)
self.gaussian_blur.weight.data[0][0] = K_BLUR.clone()34

self.batch_norm = nn.BatchNorm2d(3, affine=False) if batch_norm_output else None35
36

self.register_buffer(’M_RGB_2_YUV’, M_RGB_2_YUV.clone())
self.register_buffer(’M_YUV_2_RGB’, M_YUV_2_RGB.clone())

37
38
39

self.additive_layer = None

40

where
1
2
3

K_G = torch.Tensor([[0, 1, 0],[1, 4, 1],[0, 1, 0]]) / 4

4
5
6
7

K_RB = torch.Tensor([[1, 2, 1],[2, 4, 2],[1, 2, 1]]) / 4

8

M_RGB_2_YUV = torch.Tensor([[0.299, 0.587, 0.114],[-0.14714119, -0.28886916, 0.43601035],[0.61497538, -0.51496512, -0.10001026]])
12 M_YUV_2_RGB = torch.Tensor([[1.0000000000e+00, -4.1827794561e-09, 1.1398830414e+00],
13[1.0000000000e+00, -3.9464232326e-01, -5.8062183857e-01],
14[1.0000000000e+00, 2.0320618153e+00, -1.2232658220e-09]])
9

10
11

15

K_BLUR = torch.Tensor([[6.9625e-08, 2.8089e-05, 2.0755e-04, 2.8089e-05, 6.9625e-08],[2.8089e-05, 1.1332e-02, 8.3731e-02, 1.1332e-02, 2.8089e-05],[2.0755e-04, 8.3731e-02, 6.1869e-01, 8.3731e-02, 2.0755e-04],
19[2.8089e-05, 1.1332e-02, 8.3731e-02, 1.1332e-02, 2.8089e-05],
20[6.9625e-08, 2.8089e-05, 2.0755e-04, 2.8089e-05, 6.9625e-08]])
21 K_SHARP = torch.Tensor([[0, -1, 0],22[-1, 5, -1],
23
[0, -1, 0]])24 DEFAULT_CAMERA_PARAMS = (25[0., 0., 0., 0.],26
[1., 1., 1.],
27
[1., 0., 0., 0., 1., 0., 0., 0., 1.],
28 )
16
17
18

Note that the camera parameters are camera, and conversely in our case dataset, dependent and defined in the dataset classes.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

B.2. Description of the task models ΦTask

ΦTask

Two task models are employed in the experiments. For the classification task on the Raw-Microscopy dataset a 18layer residual net (ResNet18) [30] was used as reference task model. To segment cars from the Raw-Drone dataset the
convolutional neural network proposed in [72] (U-Net) was used. Both task models were trained using common data
augmentations on processed views v of the image measurements to avoid naive robustness failures. A detailed descriptionof the task models and their hyperparameters is given below.

Classification

Segmentation

ResNet18 based on [30]trained with Adam [39] for 100 epochs
learning rate: 10−4
mini-batch size: 128

U-Net++ based on [72]trained with Adam for 100 epochs
learning rate: 7.5 ⋅ 10−5mini-batch size: 12

Table 2: Summary of the training procedure for both task models.

Table 3: A set of random test samples for the segmentation task under learned processing. Top row: Targets, middle row:
predictions of the task model after the first epoch, last row: predictions of the task model after the last epoch.
ResNet18 This model is designed to classify images from ImageNet [76] and has therefore an output dimension of 1000. In
order to use the model to classify images from Raw-Microscopy, we changed the output dimension of the fully-connected
layer to nine. The model was trained for 100 epochs using pre-trained ResNet features. Hyperparameters were kept
constant across all runs to isolate the effect of varying image processing pipelines. For implementation the code provided
at https://pytorch.org/hub/pytorch_vision_resnet/ was used. The model consists of 34 layers withapproximately 11.2 million trainable parameters. The storage size of the model is 44.725 MB.
U-Net++ The model was trained for 100 epochs using pretrained ResNet features as the encoder of the U-Net++. Hyperparameters were kept constant across all runs to isolate the effect of varying image processing pipelines. For implementation
we used the code provided at https://github.com/qubvel/segmentation_models.pytorch. The modelData Models for Dataset Drift Controls in Machine Learning With Optical Images

has approximately 26.1 million trainable parameters. The storage size of the model is 104.315 MB.
Raw training In the drift optimization experiments of Section 2.3 the raw data is demosaiced using class
RawToRGB(nn.Module) from /processing/pipeline_torch.py in the data model code. Then task models are tuned to raw data under the same regimes described above.
For a summary of the training procedure see Table 2.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

B.3. Additional results
B.3.1. D RIFT SYNTHESIS
1 Drift synthesis with Φstat
Proc : Drone

Figure 7: 5-fold cross-validation results of the Raw-Drone drift synthesis experiments. Each cell contains the average
IoU with a color coded border for the standard deviation. Task models were trained on the data model on the vertical axis
and then tested on processed data as indicated on the horizontal axis. Numbers 1-3 left to the vertical axis denote theranking of task models according to their average IoU across all test pipelines respective corruptions. Stars denote the train
pipeline under which the task model performed best on the respective test pipeline/corruption. Full ranking results can befound in Tables 5, 8 and 9 of Appendix B.3. Left: Varying the data model leads to mixed performance drops. Diagonal isΦProc = Φ̃Proc . Right: Comparison to the corruption benchmark at medium severity (level 3). The average performance drop
is more than four times higher compared to data model variations. First column is ΦProc = Φ̃Proc . Bottom: Visual inspectionof worst case (globally worst scoring) train/test pipelines.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Microscopy
Average accuracy

Drone
Average IoU

0.75 ± 0.09
0.54 ± 0.21
0.78 ± 0.08
0.67 ± 0.14
0.75 ± 0.07

0.59 ± 0.05
0.59 ± 0.05
0.74 ± 0.04
0.71 ± 0.05
0.60 ± 0.07

Learned (low)Frozen (low)Learned (high)Frozen (high)Direct raw

Table 4: Tabular summary of the drift optimization results. The average accuracy and standard deviations over crossvalidation runs and training steps are displayed, summarizing both the stability and converge trajectory for each setting.

Rank

Microscopy-ISP
Train pipeline
Avg. score

1
2
3
4
5
6
7
8
9
10
11
12

ma,s,mema,u,mema,u,gabi,s,mebi,u,meme,s,mebi,s,game,s,game,u,mema,s,gabi,u,game,u,ga

Microscopy-CC
Train pipeline
Avg. score

0.83
0.83
0.82
0.81
0.81
0.81
0.81
0.80
0.80
0.80
0.79
0.79

bi,u,meme,s,mebi,u,gama,s,meme,u,mema,s,gama,u,meme,s,gabi,s,mema,u,gabi,s,game,u,ga

Drone-ISP
Train pipeline
Avg. score

0.63
0.63
0.62
0.62
0.62
0.62
0.61
0.60
0.59
0.59
0.58
0.58

ma,u,gabi,s,gabi,s,mema,s,meme,u,game,u,mema,u,mema,s,gabi,u,meme,s,meme,s,gabi,u,ga

Drone-CC
Train pipeline
Avg. score

0.68
0.68
0.67
0.67
0.67
0.67
0.66
0.66
0.65
0.65
0.64
0.61

ma,s,gabi,s,game,s,gama,s,meme,s,mema,u,gabi,s,mema,u,meme,u,meme,u,gabi,u,mebi,u,ga

0.60
0.57
0.57
0.55
0.55
0.55
0.54
0.54
0.53
0.51
0.48
0.46

Table 5: Rankings of task models from Section 2.1 trained on different data models (columns 2, 4, 6, 8) according totheir average accuracy or IoU (columns 3, 5, 7, 9) across all test pipelines respective corruptions. ISP corresponds to drift
synthesis with physically faithful data models, CC corresponds to common corruptions.
Rankbi,s,mebi,s,gabi,u,mebi,u,gama,s,meMicroscopy-ISP
ma,s,gama,u,mema,u,game,s,meme,s,game,u,meme,u,ga

1
2
3
4
5
6
7
8
9
10
11
12

ma,u,mema,u,gabi,s,gama,s,mebi,s,mebi,u,meme,s,meme,s,game,u,mema,s,gabi,u,game,u,gama,u,mema,u,gabi,s,gama,s,mebi,u,meme,u,mebi,s,meme,s,meme,s,gama,s,game,u,gabi,u,gama,u,gabi,s,gama,s,mema,u,meme,u,mebi,u,mebi,s,meme,s,mebi,u,gama,s,game,u,game,s,gama,u,gabi,s,gama,s,mema,u,meme,u,mebi,u,meme,s,mebi,u,gabi,s,mema,s,game,s,game,u,gama,s,mebi,s,mebi,u,gama,u,mebi,u,mema,u,game,s,mebi,s,game,s,gama,s,game,u,game,u,mema,u,game,s,gama,s,gama,s,mema,u,meme,s,meme,u,mebi,u,meme,u,gabi,s,mebi,s,gabi,u,gama,u,gama,u,mema,s,meme,u,mema,s,gabi,s,game,s,game,s,mebi,u,mebi,s,meme,u,gabi,u,gama,u,mema,s,mebi,s,game,s,gabi,u,mema,u,game,u,meme,s,mebi,s,mema,s,game,u,gabi,u,game,s,gama,u,gama,s,game,u,game,s,mema,u,mema,s,meme,u,mebi,s,mebi,s,gabi,u,mebi,u,gama,u,gama,u,mema,s,meme,u,mebi,s,game,u,game,s,mebi,s,meme,s,gama,s,gabi,u,mebi,u,gama,u,gama,u,mema,s,meme,u,mebi,s,game,u,game,s,mebi,s,meme,s,gama,s,gabi,u,mebi,u,gama,u,gama,s,mema,u,mebi,s,game,u,meme,s,game,s,mema,s,gabi,u,mebi,s,meme,u,gabi,u,ga

Table 6: Ranking of task models from Section 2.1 trained under different train pipelines (rows) for each individual test
pipeline (columns 2 - 13).
Rank

identity

gauss noise

shot

impulsespeckleMicroscopy-CC
gauss blurzoom

contrast

brightness

saturateelastic

1
2
3
4
5
6
7
8
9
10
11
12

ma,u,mema,u,gabi,s,game,s,mema,s,meme,u,meme,s,gabi,u,mebi,u,gama,s,gabi,s,meme,u,gama,u,mema,s,game,u,meme,s,gabi,u,mema,u,game,s,mebi,s,mema,s,mebi,u,gabi,s,game,u,gabi,u,mema,s,game,s,mema,u,meme,s,game,u,mebi,s,mebi,u,gama,s,mema,u,gabi,s,game,u,gabi,u,meme,u,mebi,u,game,s,mema,s,mema,u,mema,u,game,s,game,u,gama,s,gabi,s,mebi,s,gama,s,gabi,u,meme,s,meme,u,mebi,u,gama,s,mema,u,meme,s,gabi,s,mema,u,game,u,gabi,s,gabi,s,gama,u,mema,u,gabi,u,meme,u,mema,s,meme,s,gama,s,game,s,mebi,u,gabi,s,meme,u,gabi,s,gama,u,gama,s,mema,u,mebi,u,meme,s,mebi,u,game,u,game,u,meme,s,gama,s,gabi,s,mebi,s,gabi,u,game,u,gama,s,mema,u,mebi,s,mebi,u,meme,s,mema,s,gama,u,game,u,meme,s,game,s,mema,s,mebi,u,gama,s,gabi,s,mebi,u,meme,s,gama,u,game,u,gabi,s,game,u,mema,u,mema,s,meme,u,meme,s,mebi,u,gabi,s,gabi,u,mema,u,gama,s,gabi,s,meme,s,game,u,gama,u,mebi,s,gama,u,gama,u,mema,s,meme,u,meme,s,game,s,mebi,u,gabi,u,mema,s,game,u,gabi,s,meTable 7: Ranking of task models from Section 2.1 trained under different train pipelines (rows) for each individual test
corruptions (columns 2 - 12).

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Rankbi,s,mebi,s,gabi,u,mebi,u,gama,s,meDrone-ISP
ma,s,gama,u,mema,u,game,s,meme,s,game,u,meme,u,ga

1
2
3
4
5
6
7
8
9
10
11
12

bi,s,mebi,u,mema,u,gabi,s,game,u,mebi,u,gama,s,meme,u,gama,u,meme,s,mema,s,game,s,gabi,s,gabi,s,mema,u,gama,s,meme,u,gama,s,gama,u,meme,s,game,u,meme,s,mebi,u,mebi,u,gabi,u,mebi,s,mebi,u,gama,u,game,u,mebi,s,gama,u,mema,s,meme,u,game,s,meme,s,gama,s,gabi,u,mebi,s,mebi,u,gama,u,game,u,mebi,s,gama,u,mema,s,meme,u,game,s,mema,s,game,s,gama,u,gama,s,mebi,s,game,u,gama,s,gama,u,meme,u,meme,s,mebi,s,meme,s,gabi,u,mebi,u,gama,s,game,s,gama,s,meme,s,mebi,s,gama,u,game,u,game,u,mema,u,mebi,s,mebi,u,mebi,u,gama,u,game,u,mema,u,mebi,s,mema,s,mebi,s,game,u,gabi,u,mema,s,game,s,mebi,u,game,s,gama,s,mema,s,gama,u,gabi,s,game,u,game,s,meme,s,game,u,mema,u,mebi,s,mebi,u,mebi,u,gama,s,game,s,gama,s,meme,s,mebi,s,gama,u,game,u,game,u,mema,u,mebi,s,mebi,u,mebi,u,gama,u,game,u,mema,s,meme,u,gama,u,mebi,s,mebi,u,mebi,s,game,s,mebi,u,gama,s,game,s,gama,u,game,u,game,u,mema,s,mema,u,mebi,s,mebi,s,gabi,u,mema,s,game,s,mebi,u,game,s,gama,u,game,u,mema,u,mebi,s,mema,s,mebi,s,game,u,gabi,u,mebi,u,gama,s,game,s,meme,s,ga

Table 8: Ranking of task models from Section 2.1 trained under different train pipelines (rows) for each individual test
pipeline (columns 2 - 13).

Rank

identity

gauss noise

shot

impulsespeckleDrone-CC
gauss blurzoom

contrast

brightness

saturateelastic

1
2
3
4
5
6
7
8
9
10
11
12

ma,s,gabi,s,game,s,gama,s,mema,u,gabi,s,meme,u,gabi,u,mema,u,meme,u,meme,s,mebi,u,gama,s,game,s,gabi,s,game,s,mema,u,gama,u,meme,u,mema,s,mebi,s,meme,u,gabi,u,mebi,u,gama,s,game,s,gabi,s,gama,s,meme,s,mema,u,gama,u,mebi,s,meme,u,meme,u,gabi,u,mebi,u,gama,s,game,s,game,s,mebi,s,gama,u,gama,u,meme,u,mema,s,mebi,s,meme,u,gabi,u,mebi,u,gama,s,game,s,gabi,s,gama,s,meme,s,mema,u,gabi,s,mema,u,meme,u,meme,u,gabi,u,mebi,u,gama,s,gabi,s,gama,s,mema,u,gabi,u,mebi,s,meme,s,gama,u,meme,u,meme,s,meme,u,gabi,u,gabi,s,mema,s,gabi,s,game,s,gama,u,meme,s,mema,s,mebi,u,meme,u,mebi,u,gama,u,game,u,gabi,s,gama,s,game,s,mema,s,mebi,s,mema,u,mema,u,game,s,gabi,u,mebi,u,game,u,game,u,mebi,s,gama,s,gama,s,meme,s,mema,u,mema,u,game,u,mebi,s,meme,u,game,s,gabi,u,mebi,u,gama,s,gama,s,mema,u,game,u,game,s,gabi,s,gabi,s,meme,s,meme,u,mebi,u,mema,u,mebi,u,gama,s,gama,u,gama,s,mebi,s,gabi,s,mebi,u,meme,s,game,u,meme,u,game,s,mema,u,mebi,u,ga

Table 9: Ranking of task models from Section 2.1 trained under different train pipelines (rows) for each individual test
corruptions (columns 2 - 12).

Common corruptions benchmark [31]Microscopy

ISP

Figure 8: Experiment from Section 2.1 with weak severity (level 1) for the Common corruptions benchmark.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Common corruptions benchmark [31]Microscopy

ISP

Figure 9: Experiment from Section 2.1 with strong severity (level 5) for the Common corruptions benchmark.

Common corruptions benchmark [31]

Drone

ISP

Figure 10: Experiment from Section 2.1 with weak severity (level 1) for the Common corruptions benchmark.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Common corruptions benchmark [31]

Drone

ISP

Figure 11: Experiment from Section 2.1 with strong severity (level 5) for the Common corruptions benchmark.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Figure 12: A comparative overview of the physically faithful data models (ISPs, top-left) and the Common Corruptions(CC, top-right) used in the the drift synthesis experiments of Section 2.1. A matching heuristic based on possible visual
perception of the drift artifacts (top-middle) is provided for readers who would like to relate specific data models to specificcorruptions. However, we emphasize that this is a purely qualitative heuristic and has no metrological basis. Since CCs are
not physically faithful it is not clear how to relate them to actual variations in the optical data generating process. Finally,
corruptions that were excluded from the experiments in Section 2.1 are displayed (bottom). The CC examples where stitchedfrom the original paper [32] for authenticity.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Simulation of test samples
Physically faithful test samples
Differentiable data model

Augmentation testing

Catalogue testing

Data models

✓
✗
✗

✗
✓
✗

✓
✓
✓

Table 10: Comparison of empirical dataset drift validation methods. Augmentation testing enables ad-hoc test case synthesis
but lacks physical faithfulness, unlike catalogue testing. Pairing raw data with data models allows for synthesis of physicallyfaithful test cases, and differentiable data models offer novel drift controls like drift forensics and adjustments.

B.4. Related work
While physically sound data models of images have to the best of our knowledge not yet found their way into the machinelearning and dataset drift literature, they have been studied in other disciplines, in particular physical optics and metrology.
Our ideas on data models and dataset drift controls we present in this manuscript are particularly indebted to the following
works.
Data models for images [87; 69] employ deep convolutional neural networks for modelling a raw image data processing
which is optimized jointly with the task model. In contrast, we employ a parametric data model with tunable parameters thatenables the modular drift forensics and synthesis presented later. [94] propose a differentiable image processing pipeline forthe purpose of camera lens manufacturing. Their goal, however, is to optimize a physical component (lens) in the image
acquisition process and no code or data is publicly available. Existing software packages that provide low level image
processing operations include Halide [68], Kornia [70] and the rawpy package [81] which can be integrated with our Python
and PyTorch code. We should also take note that outside optical imaging there are areas in machine learning and appliedmathematics, in particular inverse problems such as magnetic resonance imaging (MRI) or computed tomography, that makeuse of known operator learning [51; 49] to incorporate the forward model in the optimization [14] or, as in the case of MRI,
learn directly in the k-space [102].
Drift synthesis As detailed in the introduction, the synthesis of realistic drift test cases for a task model in computer vision isoften done by applying augmentations directly to the input view v GC , e.g. a processed .jpeg or .png image. Hendrycks
et al. [31] have done foundational work in this direction developing a practical, standardized benchmark. However, there
is no guarantee that noise ξ added to a processed image v will be physically faithful, i.e. that v + ξ ∈ Φ̃Proc [XRAW ].
This is problematic, as nuances matter [21] for assessing the cascading effects data models have on the task model ΦTask
downstream [5; 77]. For the same reason, the use of generative models [25] like GANs has been limited for test data
generation as they are known to hallucinate visible and less visible artifacts [17; 80]. Other approaches, like the WILDS
data catalogue [9; 41], build on manual curation of so called natural distribution shifts, or, like [84], on artificial worst caseconstructions. These are important tools for the study of dataset drifts, especially those that are created outside the camera
image signal processing. Absent explicit, differentiable data models and raw sensor data, the shared limitation of catalogue
approaches is that metrologically faithful drift synthesis is not possible and the data generating process cannot be granularly
studied and manipulated.
Drift forensics Phan et al. [65] use a differentiable raw processing pipeline to propagate the gradient information back to theraw image. Similar to this work, the signal is used for adversarial search. However, Phan et al. optimize adversarial noise on
a per-image basis in the raw space xRAW , whereas our work modifies the parameters of the data model ΦProc itself in pursuitof harmful parameter configurations. The goal in this work is not simply to fool a classifier, but to discover failure modes
and susceptible parameters in the data model ΦProc that will have the most influence on the task model’s performance.
Drift optimization An explicit and differentiable image processing data model allows joint optimization together withthe task model ΦProc . This has been done for radiology image data [71; 86; 50] though the measurement process there
is different and the focus lies on finding good sampling patterns. For optical data, a related strand of work is modelling
inductive biases in the image acquisition process which is explained and contrasted to this work above [94; 35].
Raw image data Camera raw files contain the data captured by the camera sensors [7]. In contrast to processed formats
such as .jpeg or .png, raw files contain the sensor data with minimal processing [96; 58; 46]. The processing of the raw
data usually differs by camera manufacturer thus contributing to dataset drift. Existing raw data sets from the machinelearning, computer vision and optics literature can be organized into two categories. First, datasets that are sometimes

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

treated - usually not by the creators but by users of the data - as raw data but which are in fact not raw. Examples for thiscategory can be found for both modalities considered here [11; 18; 3; 33; 12; 95; 22; 48; 53; 82; 103]. All of the preceding
examples are processed and stored in formats including .jpeg, .tiff, .svs, .png, .mp4 and .mov. Second, datasetsthat are labelled raw data which are raw. In contrast to the labelled and precisely calibrated raw data presented here, existing
raw datasets [19; 15; 1; 29] are collected from various sources for image enhancement tasks without full specification of themeasurement conditions or labels for classification or segmentation tasks.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Figure 13: Processed samples and labels of the two datasets, Raw-Microscopy (columns one to four) and Raw-Drone(columns five and eight), that were acquired for the dataset drift study presented here.

B.5. Datasets details
B.5.1. DATA ACQUISITIONIn the following, core information on the two acquired datasets is provided. In Appendix B.5.2 you can also find detailed
datasheets for both datasets, following the documentation good practices introduced by [24].
Raw-Microscopy Assessment of blood smears under a light microscope is a key diagnostic technique for many healthcare
services such as cancer treatment and kidney failure as well as blood disorder detection [6]. The creation of image datasetsand machine learning models on them has received wide interest in recent years [43; 56; 4]. Variations in the image
processing can affect the downstream task model performance [91]. Dataset drift controls can thus help to specify theperimeter of safe application for a task model. A raw dataset was collected for that purpose. A bright-field microscope wasused to image blood smear cytopathology samples. The light source is a halogen lamp equipped with a 0.55 NA condenser,
and a pre-centred field diaphragm unit. Filters at 450 nm, 525 nm and 620 nm were used to acquire the blue, green and redchannels respectively. The condenser is followed by a 40× objective with 0.95 NA (Olympus UPLXAPO40X). Slides canbe moved via a piezo with 1 nm spatial resolution, in three directions. Focus was achieved by maximizing the variance ofthe pixel values6 . Images are acquired at 16 bit, with a 2560 × 2160 pixels CMOS sensor (PCO edge 5.5). The point-spreadfunction (PSF) was measured to be 450 nm with 100 nm nanospheres. Mechanical drift was measured at 0.4 pixels perhour. Imaging was performed on de-identified human blood smear slides (Ma190c Lieder, J. Lieder GmbH & Co. KG,
Ludwigsburg/Germany). All slides were taken from healthy humans without known hematologic pathology. Imaging regions
were selected to contain single leukoytes in order to allow unique labelling of image patches, and regions were cropped to
256 × 256 pixels. All images were annotated by a trained hematological cytologist using the standard scheme of normal
leukocytes comprising band and segmented neutrophils, typical and atypical lymphocytes, monocytes, eosinophils andbasophils [47]. To soften class imbalance, candidates for rare normal leukocyte types were preferentially imaged, and enrichrare classes. Additionally, two classes for debris and smudge cells, as well as cells of unclear morphology were included.
Labelling took place for all imaged cells from a particular smear at a time, with single-cell patches shown in random order.
Raw images were extracted using JetRaw Data Suite features. Blue, red and green channels are metrologically rescaled
independently in intensity to simulate a standard RGB camera condition. Some pixels are discarded complementary on eachchannel in order to obtain a Bayer filter pattern.
Raw-Microscopy for segmentation comes with 940 raw images, twelve differently processed variants totaling 11280 images
and six additional raw intensity levels totaling 5640 samples.
Raw-Drone Automated processing of drone data has useful applications including precision agriculture [42] or environmental protection [36]. Variation in image processing has been shown to affect task model performance [52; 96], underlining
the need for drift controls. For the purposes of this study, a raw car segmentation dataset was created for the drone image
modality. A DJI Mavic 2 Pro Drone was used, equipped with a Hasselblad L1D-20c camera (Sony IMX183 sensor)
having 2.4 µm pixels in Bayer filter array. The lens has a focal length of 10.3 mm. The f-number was set to N = 8, toemulate the PSF circle diameter relative to the pixel pitch and ground sampling distance (GSD) as would be found on
images from high-resolution satellites. The PSF was measured to have a circle diameter of 12.5 µm. This corresponds to adiffraction-limited system, within the uncertainty dominated by the wavelength spread of the image. Images were taken at
200 ISO, a gain of 0.528 DN/e− . The 12-bit pixel values are however left-justified to 16-bits, so that the gain on the 16-bit
6Figure 14 in Appendix B.5.1 provides an illustration of the imaging setup.

Data Models for Dataset Drift Controls in Machine Learning With Optical Images

Figure 14: (a) An illustration of the imaging setup. (b) Datasets visualization. (Top-left) RGB raw microscopy classes areshown. (Top-right) Drone raw images are shown with the segmentation mask applied over it. (Bottom) Different intensity
realizations are shown for the microscopy case. Images on the top are directly print out in the same scale of the original
image. Images in the bottom row are normalized on their own min and max values to highlight the role of noise levels on
low intensity images.
Composition of Raw-Microscopy
Type of instances
Objects on images
Type of classes
Number of instances
Number of classes
Image size
Image format
Raw image format

Image and label
White blood cells
Morphological classes
940
9
256 by 256 pixels.tif
Please see Section 1.1

Class

Proportion in %Basophil (BAS)
Eosinophil (EOS)
Smudge cell / debris (KSC)
atypical Lymphocyte (LYA)
typical Lymphocyte (LYT)
Monocyte (MON)Neutrophil (band) (NGB)
Neutrophil (segmented) (NGS)Image that could not be assigned a class (UNC)

1.91
5.74
17.34
3.19
24.47
20.32
0.85
22.98
3.19

Composition of Raw-Drone
Type of instances
Objects on images
Number of instances
Number of original images
Image size
Mask size
Original image size
Image format
Mask format
Raw image format

Image and mask
Landscape shots from above548
12
256 by 256 pixels
256 by 256 pixels
3648 by 5472
.tif
.png
.DNG

Table 11: Summaries of the compositions of Raw-Microscopy and Raw-Drone
numbers is 8.448 DN/e− . The images were taken at a height of 250 m, so that the GSD is 6 cm. All images were tiled in
256 × 256 patches. Segmentation masks were created to identify cars for each patch. From this mask, classification labels
were generated to detect if there is a car in the image. The dataset is constituted by 548 images for the segmentation task.
Raw-Drone for segmentation comes with 548 raw images, twelve differently processed variants totaling 6576 images andsix additional raw intensity levels totaling 3288 samples.

B.5.2. DATASHEETS
We follow the datasheets documentation framework proposed in [24], using the template https://de.overleaf.
com/latex/templates/datasheet-for-dataset-template/jgqyyzyprxth from Christian Garbin.
Datasheet for Raw-Microscopy

Motivation
For what purpose was the dataset created?
With Raw-Microscopy we provide a publicly available raw
image dataset in order to examine the effect of the image
signal processing on the performance and the robustnessof machine learning models. This dataset enables to study
these effects for a supervised multiclass classification task:
the classification of white blood cells (WBCs).
Who created this dataset (e.g., which team, researchgroup) and on behalf of which entity (e.g., company,
institution, organization)?

larger set?
The dataset does not contain all possible instances. It islimited to WBC classes normally present in the peripheral
blood of healthy humans. In order to cope with intrinsic
class imbalance in cell distribution, rare cell class candidates
such as Basophils were preferentially imaged.
What data does each instance consist of? “Raw” data
(e.g., unprocessed text or images) or features?
Each instance consists of an image of 256 by 256 pixels.
The image is a raw image in .tiff format.
Is there a label or target associated with each instance?

This dataset has been created by the Laboratory of Applied
Optics of the Micro-Nanotechnology group at HEPIA/HESSO, University of Applied Sciences of Western Switzerland.
Single-cell images were annotated by a trained cytologist.

Each instance is associated to a label, that indicates themorphological class of the image.

Who funded the creation of the dataset?

No information is missing.

The creation of the dataset has been funded by HEPIA/HESSO.

Composition
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)?
An instance is a tuple of an image and a label. The image
shows a human WBCs and the label indicates the morphological class of this cell. The following eight morphologicalclasses appear in the dataset: Basophil (BAS), Eosinophil(EOS), Smudge cell / debris (KSC), atypical Lymphocyte(LYA), typical Lymphocyte (LYT ), Monocyte (MON), Neutrophil (band) (NGB), Neutrophil (segmented) (NGS). The
nith class consists of images that could not be assigned aclass (UNC) during the labeling process.
How many instances are there in total (of each type, if
appropriate)?
The data set consists of 940 instances. For the proportion ofeach class in the dataset see table 12.
Does the dataset contain all possible instances or is ita sample (not necessarily random) of instances from a

Is any information missing from individual instances?

Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)?
No, relationships between individuals are not made explicit.

Are there recommended data splits (e.g., training, development/validation, testing)?
There are no recommended data splits. All the data splitsthat we used for our experiments were randomly picked.
Are there any errors, sources of noise, or redundancies
in the dataset?
To the best of our knowledge, there are no errors in thedataset. However, a key source of variability between slides
from different laboratories and processing times is stain intensity. The samples used in this work all come from thesame source, hence we assume the preanalytic treatment andstaining protocol to be similar. As all images were obtainedon the same microscopy equipment, focus handling andillumination are identical for all samples. Image labelling
was performed by one trained morphologist with experience in hematological routine diagnostics. It is known thatmorphology annotations are subject to inter- and intra-ratervariability. However, as we limit ourselves to normal WBCs
the labeling is expected to be stable.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets,
other datasets)?
The dataset is self-contained.
Does the dataset contain data that might be considered
confidential (e.g., data that is protected by legal privilege
or by doctor-patient confidentiality, data that includes
the content of individuals non-public communications)?
The dataset consist of medical data, disclosing the morphological classes of single human WBCs. In principle, thedistribution of cell types conveys information on the healthstate of a patient.
However, the subjects in this dataset are fully de-identified,so that the image data cannot be linked back to the healthy
donors of the scanned blood smears. Furthermore, it is notdisclosed which cell image was taken from which bloodsmear, so that no frequencies of individual cell types canbe determined. Additionally, we only consider cell types
present in normal blood, so that no specific hematologic
pathology can be deduced from cell morphologies.
Does the dataset contain data that, if viewed directly,
might be offensive, insulting, threatening, or might otherwise cause anxiety?
No. The dataset does not contain data with any of the above
properties.

or health data; biometric or genetic data; forms of government identification, such as social security numbers;
criminal history)?
No. While the distribution of cell types for a specific patient could reveal information about that patient’s healthstatus, isolated single-cell images of normal leukocytes do
not allow for this inference.
Any other comments?
See table 13 for a summary of the composition of RawMicroscopy.
Class

Proportion in %Basophil (BAS)
Eosinophil (EOS)
Smudge cell / debris (KSC)
atypical Lymphocyte (LYA)
typical Lymphocyte (LYT)
Monocyte (MON)Neutrophil (band) (NGB)
Neutrophil (segmented) (NGS)Image that could not be assigned a class (UNC)

1.91
5.74
17.34
3.19
24.47
20.32
0.85
22.98
3.19

Table 12: The proportion of the classes in Raw-Microscopy.

Collection Process
How was the data associated with each instance acquired?
Images of the dataset have been acquired directly from a
CMOS imaging sensor. They are in a raw unprocessed
format.

Does the dataset relate to people?
Yes. The dataset consist of images of human WBCs.
Does the dataset identify any subpopulations (e.g., byage, gender)?
The donors of the blood smears used in this dataset are fully
deidentified, and no information on subpipulation composition is provided.
Is it possible to identify individuals (i.e., one or morenatural persons), either directly or indirectly (i.e., in
combination with other data) from the dataset?

What mechanisms or procedures were used to collectthe data (e.g., hardware apparatus or sensor, manualhuman curation, software program, software API)?
Imaging data have been obtained via a custom brightfieldmicroscope.
If the dataset is a sample from a larger set, what was the
sampling strategy (e.g., deterministic, probabilistic withspecific sampling probabilities)?

No. It is not possible to identify individuals from an imageof their white blood cells or visa versa.

Images have 256 × 256 pixel size and have been cropped
from larger images. The dataset corresponds to a selectionof white blood cells in the acquired large images. A sampling strategy aimed at increasing the proportion of rareclasses of white blood cells has been used.

Does the dataset contain data that might be consideredsensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, politicalopinions or union memberships, or locations; financial

Who was involved in the data collection process (e.g.,
students, crowdworkers, contractors) and how were
they compensated (e.g., how much were crowdworkerspaid)?

A research assistant has been involved in the data collection
process and has been compensated with a monthly salary.
Over what timeframe was the data collected? Does thistimeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news
articles)?
Data have been collected on a timeframe of two months,
corresponding to the availability of the physical samples toimage. Data have been collected on purpose for this work.

Were any ethical review processes conducted (e.g., by an
institutional review board)?
The microscopy data was purchased from a commercial lab
vendor (J. Lieder GmbH & Co. KG, Ludwigsburg/Germany)
who attained consent from the subjects included.
Does the dataset relate to people?
Yes. The dataset consists of microscopic images of humanwhite blood cells.
Did you collect the data from the individuals in question
directly, or obtain it via third parties or other sources(e.g., websites)?

Preprocessing/cleaning/labeling
Was any preprocessing/cleaning/labeling of the data
done (e.g., discretization or bucketing, tokenization, partof-speech tagging, SIFT feature extraction, removal of
instances, processing of missing values)?
Intensity scaled images are generated with Jetraw Data Suitefor both datasets, which applies a physical model based on
sensor calibration to accurately simulate intensity reduction. Microscopy Raw images are extracted from RGB Microscopy data through a pixel selection from images takenwith three filters, in order to have a Bayer Pattern. Pixels
intensities are rescaled with Jetraw Data Suite to match themeasured transmissivities of a Bayer colour filters array.
Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)?
Raw images are available in the dataset.
Is the software used to preprocess/clean/label the instances available?
All code used in the experiments of this manuscript is publicly available. Jetraw products that were used for acquiring
the data are commercially available.

Data have not been obtained via third parties.
Were the individuals in question notified about the datacollection?
As the blood smear slides were bought from a company,
notification to individuals of the data collection has been
performed by the company.
Did the individuals in question consent to the collection
and use of their data?

Uses
Has the dataset been used for any tasks already?
The dataset has not yet been used.
Is there a repository that links to any or all papers or
systems that use the dataset?

Yes, they did.

The
repositoryathttps://github.com/
aiaudit-org/raw2logit associated to this work,
maintained by Luis Oala.

If consent was obtained, were the consenting individuals
provided with a mechanism to revoke their consent in
the future or for certain uses?

What (other) tasks could the dataset be used for?

We do not know the conditions of consent adopted by the
selling company. However, we believe the company provided the individuals a complete freedom in revoking theirconsent in the future, if desired.
Has an analysis of the potential impact of the dataset andits use on data subjects (e.g., a data protection impact
analysis) been conducted?
No, this kind of analysis has not been conducted.

The dataset can be used to study the effect of image signal
processing on the performance and robustness of any othermachine learing model implemented in PyTorch, designed
for a supervised multiclass classification task.
Is there anything about the composition of the dataset
or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?
To the best of our knowledge, we do not recognize suchimpacts.

Are there tasks for which the dataset should not be used?

Who will be supporting/hosting/maintaining thedataset?

To the best of our knowledge, there are no such tasks.

Luis Oala on behalf of Dotphoton AG.

Distribution
Will the dataset be distributed to third parties outsideof the entity (e.g., company, institution, organization) on
behalf of which the dataset was created?
Yes. The dataset will be publicly available.
How will the dataset will be distributed (e.g., tarball on
website, API, GitHub)
A guide to access the dataset is available at https://github.com/aiaudit-org/raw2logit. Moreover, the dataset can be downloaded anonymously and directly at https://zenodo.org/record/5235536
under the doi: 10.5281/zenodo.5235536.
When will the dataset be distributed?
The dataset is already publicly available.
Will the dataset be distributed under a copyright orother intellectual property (IP) license, and/or underapplicable terms of use (ToU)?
The dataset will be distributed under the Creative Commons
Attribution 4.0 International.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances?
No.
Do any export controls or other regulatory restrictions
apply to the dataset or to individual instances?
There are no such restrictions.
MaintenanceHow can the owner/curator/manager of the dataset becontacted (e.g., email address)?
By email address via
luis.oala@dotphoton.com.
Is there an erratum?
At the time of submission, there is no such erratum. If an erratum is needed in the future it will be accessible at https://github.com/aiaudit-org/raw2logit.
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?
Yes. The dataset will be enlarged wrt. the number of instances.
If the dataset relates to people, are there applicable limits
on the retention of the data associated with the instances(e.g., were individuals in question told that their data
would be retained for a fixed period of time and thendeleted)?
To the best of our knowledge, there are no such limits.
Will older versions of the dataset continue to be supported/hosted/maintained?
Older versions will be supported and maintained in the future. The dataset will continue to be hosted as long as
https://zenodo.org/ exists.
If others want to extend/augment/build on/contribute to
the dataset, is there a mechanism for them to do so?
For any of these requests contact either Luis
Oala (luis.oala@dotphoton) or Bruno Sanguinetti(bruno.sanguinetti@dotphoton.com). For now, we do nothave an established mechanism to handle these requests.

Composition of Raw-Microscopy
Type of instances
Objects on images
Type of classes
Number of instances
Number of classes
Image size
Image format
Raw image format

Image and label
White blood cells
Morphological classes
940
9
256 by 256 pixels.tif
Please see Section 1.1

Table 13: A summary of the composition of RawMicroscopy.

Datasheet for Raw-Drone

Motivation
For what purpose was the dataset created?
With Raw-Drone we provide a publicly available raw dataset
in order to examine the effect of the image data processing
on the performance and the robustness of machine learningmodels. This dataset enables to study these effects for a segmentation task: the segmentation of cars. The dataset was
taken with specified parameters: sensor gain, point-spreadfunction and ground-sampling distance, so that physicalmodels may be used to process the data. It also was takenwith a easily accessible and affordable system, so that it
may be reproduced.
Who created this dataset (e.g., which team, researchgroup) and on behalf of which entity (e.g., company,
institution, organization)?
The dataset was created by Bruno Sanguinetti and Marco
Aversa on behalf of the company Dotphoton AG.
Who funded the creation of the dataset?
The data collection was funded by Dotphoton AG. The calibration of the image characteristics was jointly funded by
Dotphoton AG and the European Space Agency.
Composition
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)?
An instance is a tuple of an image and a segmentation mask.
The image shows a landscape shot from above. The segmentation mask is a binary image. A white pixel in this maskcorresponds to a pixel within a region in the image where acar is displayed. A black pixel in this mask corresponds to a
pixel within a region in the image where no car is displayed.

How many instances are there in total (of each type, if
appropriate)?

What data does each instance consist of? “Raw” data
(e.g., unprocessed text or images) or features?
Both, the image and the segmentation mask consist of 256
by 256 pixels. The image is a raw image in .tif format
and the the segmentation mask is in .png format. The
images are cropped sub-images of 12 raw images in .DNG
format, consisting of 3648 by 5472 pixels.
Is there a label or target associated with each instance?
Each instance is associated to a binary segmentation mask.

Is any information missing from individual instances?
No information is missing.
Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)?
Since every image is a cropped sub-image of an original image, several of these sub-images belong to the same original
image. All sub-images are disjoint, i.e. no different images
share a pixel from the original image.
Are there recommended data splits (e.g., training, development/validation, testing)?
There are no recommended data splits. All the data splitsthat we used for our experiments were randomly picked.
Are there any errors, sources of noise, or redundancies
in the dataset?
To the best of our knowledge, there are no errors in thedataset. The segmentation mask is created by hand andhence noisy, especially at the boundaries between a region
with a car and a region without a car.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets,
other datasets)?
The dataset is self-contained.

The dataset consists of 548 instances.
Does the dataset contain all possible instances or is ita sample (not necessarily random) of instances from alarger set?

Does the dataset contain data that might be considered
confidential (e.g., data that is protected by legal privilege
or by doctor-patient confidentiality, data that includes
the content of individuals non-public communications)?

The dataset does not contain all possible instances. Only
images with at least one white pixel in the associated segmentation mask are considered.

No. The dataset does not contain data of any of the above
types.

Does the data set contain data that, if viewed directly,
might be offensive, insulting, threatening, or might otherwise cause anxiety?
No. The dataset does not contain data with any of the above
properties.
Does the dataset relate to people?
The dataset does not relate to people. The drone data wasscreened for PIIs such as faces or license plates on cars andremoved by the data collection team.
Any other comments?

6 cm. All images were tiled in 256x256 patches. Segmentation color masks were created to identify cars for eachpatch. From this mask, classification labels were generated
to detect if there is a car in the image. The dataset is constituted by 548 images for the segmentation task, and 930 for
classification. Six additional intensity scales were createdwith Jetraw.
If the dataset is a sample from a larger set, what was the
sampling strategy (e.g., deterministic, probabilistic withspecific sampling probabilities)?
The entire dataset is presented.

See table 14 for a summary of the composition of the RawDrone.

Who was involved in the data collection process (e.g.,
students, crowdworkers, contractors) and how were
they compensated (e.g., how much were crowdworkerspaid)?

Collection Process

The dataset was taken by a company employee, compensated by his salary. Labeling was performed by both a company employee and a PhD student, who’s PhD is funded by
the company.

How was the data associated with each instance acquired?
The data was collected by flying a drone and saving the raw
data. The calibration data for the drone’s imager was acquired both under laboratory conditions and using a groundbased calibration target, so that it could be acquired underoperating conditions.
What mechanisms or procedures were used to collectthe data (e.g., hardware apparatus or sensor, manualhuman curation, software program, software API)?
To acquire the drone images, we used a DJI Mavic 2 Pro
Drone, equipped with a Hasselblad L1D-20c camera (SonyIMX183 sensor). This system has 2.4 µm pixels in Bayerfilter array. Images were taken with the drone hovering
for maximum stability. This stability was verified to bebetter than a single pixel by calculating the correlation ofsubsequent images. The objective has a focal length of
10.3 mm. We operated this objective at an f-number of
N = 8, to emulate the PSF circle diameter relative to the
pixel pitch and ground sampling distance (GSD) as would
be found on images from high-resolution satellites. Operating at N = 8 also minimises vignetting, aberrations, andincreases depth of focus. The point-spread function (PSF)
was measured to have a circle diameter of 12.5 µm using theedge-spread function technique and a ground calibration target.This corresponds to σ = 2.52 px, which also corresponds
to a diffraction-limited system, within the uncertainty dictated by the wavelength spread of the image. Images weretaken at 200 ISO, corresponding to a gain of 0.528 DN/e− .
The 12-bit pixel values are however left-justified to 16-bits,so that the gain on the 16-bit numbers is 8.448 DN/e− . The
images were taken at a height of 250 m, so that the GSD is

Over what timeframe was the data collected? Does thistimeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news
articles)?
The dataset was taken as the initial step of writing thisarticle.
Were any ethical review processes conducted (e.g., by an
institutional review board)?
The dataset does not contain any elements requiring an
ethical review process.
Does the dataset relate to people?
The dataset does not relate to people. There are individuals on the images, but it is not possible to identify these
individuals.
Preprocessing/cleaning/labeling
Was any preprocessing/cleaning/labeling of the data
done (e.g., discretization or bucketing, tokenization, partof-speech tagging, SIFT feature extraction, removal of
instances, processing of missing values)?
No further processing was applied to the Raw-Drone data.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)?

Raw images are available in the dataset.

When will the dataset be distributed?

Is the software used to preprocess/clean/label the instances available?

The dataset is already publicly available.

All code used in the experiments of this manuscript
is publicly available. Jetraw products that were used
for acquiring the data are commercially available.
Uses

Has the dataset been used for any tasks already? The
dataset has not yet been used.
Is there a repository that links to any or all papers or
systems that use the dataset?
Therepositoryathttps://github.com/
aiaudit-org/raw2logit associated to this work,
maintained by Luis Oala.

Will the dataset be distributed under a copyright orother intellectual property (IP) license, and/or underapplicable terms of use (ToU)?
The dataset will be distributed under the Creative Commons
Attribution 4.0 International.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances?
No.
Do any export controls or other regulatory restrictions
apply to the dataset or to individual instances?
There are no such restrictions.

What (other) tasks could the dataset be used for?

MaintenanceThe dataset can be used to study the effect of image signal
processing on the performance and robustness of any othermachine learing model implemented in PyTorch, designed
segmentation task.

Who will be supporting/hosting/maintaining thedataset?

Is there anything about the composition of the dataset
or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?

How can the owner/curator/manager of the dataset becontacted (e.g., email address)?

To the best of our knowledge, we do not recognize suchimpacts.

By email address via
luis.oala@dotphoton.com.

Are there tasks for which the dataset should not be used?

Is there an erratum?

To the best of our knowledge, there are no such tasks.

At the time of submisson, there is no such erratum. If an erratum is needed in the future it will be accessible at https://github.com/aiaudit-org/raw2logit.

Luis Oala on behalf of Dotphoton AG.

Distribution

Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?

Will the dataset be distributed to third parties outsideof the entity (e.g., company, institution, organization) on
behalf of which the dataset was created?

Yes. The dataset will be enlarged wrt. the number of instances.

Yes. The dataset will be publicly available.

If the dataset relates to people, are there applicable limits
on the retention of the data associated with the instances(e.g., were individuals in question told that their data
would be retained for a fixed period of time and thendeleted)?

How will the dataset will be distributed (e.g., tarball on
website, API, GitHub)
A guide to access the dataset is available at https://github.com/aiaudit-org/raw2logit. Moreover, the dataset can be downloaded anonymously and directly at https://zenodo.org/record/5235536
under the doi: 10.5281/zenodo.5235536.

To the best of our knowledge, there are no such limits.
Will older versions of the dataset continue to be supported/hosted/maintained?

Older versions will be supported and maintained in the future. The dataset will continue to be hosted as long as
https://zenodo.org/ exists.
If others want to extend/augment/build on/contribute toComposition of Raw-Drone
Type of instances
Objects on images
Number of instances
Number of original images
Image size
Mask size
Original image size
Image format
Mask format
Raw image format

Image and mask
Landscape shots from above548
12
256 by 256 pixels
256 by 256 pixels
3648 by 5472
.tif
.png
.DNG

Table 14: A summary of the composition of Raw-Drone.

the dataset, is there a mechanism for them to do so?
For any of these requests contact either Luis Oala(luis.oala@dotphoton.com) or Bruno Sanguinetti(bruno.sanguinetti@dotphoton.com). For now, we do nothave an established mechanism to handle these requests.