title: Empirical perspectives on deep learning
title: from scaling and generalization to new compute paradigms
titleType: Subtitle
Creators: name: Becker, Marlon
Publisher: name: University of Muenster, Institutional Repository MIAMI
Year: 2025
Subjects: subject: 000 Computer science, information, general works

description: Die durch neue Rechentechnologien ermöglichte Skalierung künstlicher neuronaler Netze (ANNs) auf nie dagewesene Größen hat nicht nur zu quantitativen Verbesserungen geführt, sondern auch zu qualitativ neuen Phänomenen, die nur bei starker Überparametrisierung zu beobachten sind.
Mit traditionellen theoretischen Ansätzen lassen sich diese großen Modelle nur schwer erfassen.
Um dieses Problem anzugehen, werden in dieser Arbeit empirische Methoden angewandt, um die komplexe, scheinbar chaotische Dynamik großer ANNs zu untersuchen.
Es werden zwei empirische Ansätze vorgestellt, um Generalisierung in ANNs besser zu verstehen und zu verbessern.
Der erste Ansatz untersucht die Breite von Minima in der Loss-Landschaft von ANNs, während der zweite Ansatz die Speicherung einzelner Stichproben durch Vorhersagen zufällig generierter Labels untersucht.
Darüber hinaus haben sich Optische Neuronale Netze (ONNs) als vielversprechende Alternative zu herkömmlichen elektronischen digitalen Rechenarchitekturen herausgestellt.
In dieser Arbeit werden mehrere Strategien für die Implementierung von ONNs vorgeschlagen und es wird erörtert, wie die spezifischen physikalischen Beschränkungen von ONNs aus Sicht der Informatik neue Fragen aufwerfen.
description: Enabled by new computing technologies, the scaling of Artificial Neural Networks (ANNs) to unprecedented sizes has led to more than just quantitative improvements - qualitatively new phenomena have emerged, observable only under heavy overparameterization.
Traditional theoretical approaches struggle to fully capture these large-scale models.
To address this issue, this thesis adopts empirical methods to investigate the complex, seemingly chaotic dynamics of large ANNs.
Two empirical approaches are presented to better understand and improve generalization.
The first approach examines the sharpness of minima in the loss landscape of ANNs, while the second investigates the memorization of individual samples utilizing predictions of randomly generated labels.
Additionally, Optical Neural Networks (ONNs) have emerged as a promising alternative to traditional electronic digital computing architectures.
This thesis proposes several strategies for implementing ONNs and discusses how the specific physical constraints of ONNs raise new questions from a computer science perspective.
description: 0 On the Mutual Relation of the Natural Sciences and Deep
Learning ..... 1
0.1 Deep Learning for Natural Sciences ..... 1
0.2 Natural Sciences for Deep Learning ..... 2
1 Introduction ..... 5
Part i Foundational Concepts
2 Artificial Neural Networks ..... 15
2.1 Multilayer perceptron ..... 15
2.2 Activation Functions ..... 16
2.3 ANN training ..... 17
2.3.1 Gradient Based Optimization ..... 18
2.4 Convolutional Layers ..... 20
2.4.1 Pooling ..... 20
2.5 Vanishing Gradient ..... 21
2.6 Normalization Layers ..... 21
2.7 Skip Connections ..... 22
2.8 Transformer ..... 22
2.8.1 Vision Transformer ..... 23
2.9 Regularization ..... 24
2.9.1 Weight Decay ..... 24
2.9.2 Label Smoothing ..... 24
2.9.3 Dropout ..... 25
3 Classical Learning Theory and Its Limitations ..... 27
3.1 Universal Approximation Theorem ..... 27
3.2 Overfitting and Bias-Variance Tradeoff ..... 28
3.3 PAC Learning ..... 29
3.4 Rademacher Complexities ..... 31
3.5 Vapnik-Chervonenkis dimension ..... 32
3.6 Double Descent ..... 33
4 Markov Chain Monte Carlo ..... 35
4.1 Metropolis-Hastings Algorithm ..... 36
4.2 Metropolis Algorithm ..... 37
4.3 Simulated Annealing ..... 38
5 Reinforcement Learning ..... 41
5.1 Markov Decision Processes ..... 42
5.2 Value Functions ..... 43
5.3 Temporal Difference Learning ..... 44
5.3.1 SARSA ..... 44
5.3.2 Q-Learning ..... 45
5.4 Deep Reinforcement Learning ..... 45
6 Optical Neural Networks ..... 47
6.1 Analog Computing ..... 47
6.2 Optical Signal Encoding ..... 48
6.3 Phase Change Materials for Matrix-Vector Multiplication ..... 49
Part ii Generalization in Deep Learning
7 Momentum-SAM ..... 53
7.1 Introduction ..... 53
7.2 Related Work ..... 56
7.3 Method ...... 57
7.3.1 Notation ..... 57
7.3.2 Sharpness Aware Minimization (SAM) ..... 57
7.3.3 Momentum and Nesterov Accelerated Gradient ..... 58
7.3.4 Momentum-SAM ..... 59
7.4 Evaluating MSAM ..... 62
7.4.1 Speed and Accuracy for ResNets on CIFAR100 ..... 62
7.4.2 ResNet and ViT on ImageNet ..... 63
7.4.3 Combination with other SAM Variants ..... 64
7.4.4 Comparison with Same Computational Budgets ..... 64
7.4.5 Machine Translation ..... 65
7.4.6 Effects of MSAM/SAM During ViT’s Warm-up
Phase ..... 66
7.5 Properties of MSAM ..... 66
7.5.1 Negative and Positive Perturbation Directions ..... 67
7.5.2 Overshooting of Local Minima ..... 69
7.5.3 Similarity between SAM and MSAM ..... 73
7.5.4 Loss Sharpness Analysis ..... 74
7.5.5 Curvature ..... 76
7.5.6 Loss Ascent in Momentum Direction ..... 77
7.5.7 Hyperparemeter Stability ..... 77
7.5.8 Feature Rank Analysis ..... 78
7.6 Theoretical Analysis of MSAM ..... 78
7.7 Ablation Studies on MSAM ..... 80
7.7.1 Normalization ..... 80
7.7.2 Random and Last Gradient Perturbations ..... 81
7.8 Conclusion ..... 82
8 Learned Random Label Predictions as a Neural Network Complexity
Metric ..... 83
8.1 Introduction ..... 83
8.2 Related Work ..... 86
8.3 Notation ...... 87
8.4 Methods ...... 87
8.5 Experiments ..... 89
8.5.1 Complexity Metric ..... 89
8.5.2 Regularization ..... 92
8.5.3 Relation to Label Smoothing ..... 99
8.5.4 Ablation ..... 101
8.6 Conclusion ..... 103
Part iii Optical Neural Networks
9 Activation Functions in Non-Negative Neural Networks ..... 109
9.1 Introduction ..... 109
9.2 Non-Negative Activation Functions ..... 111
9.3 Sequential Weight Initialization ..... 112
9.4 Evaluating Activation Functions in 4Ns ..... 114
9.5 Conclusion ..... 116
10 Physical Chemical Activation Functions ..... 119
10.1 Introduction ..... 119
10.2 Contribution ..... 121
10.3 Design and implementation of the PCAF ..... 121
10.4 PCAF modelling and fitting ..... 123
10.5 Training neural networks using the PCAF ..... 128
10.6 Neural Network Training Details ..... 131
10.7 Conclusion ..... 132
11 Memory Metropolis and Template Networks ..... 135
11.1 Introduction ..... 135
11.2 Related Work ..... 139
11.3 Methods ...... 140
11.3.1 Memory Metropolis ..... 140
11.3.2 Template Networks ..... 144
11.4 Simulation Results ..... 147
11.4.1 Optimization Process ..... 148
11.4.2 Comparison against SA ..... 148
11.4.3 Comparison against and regularized SA ..... 151
11.5 Properties of MeMe ..... 154
11.5.1 Effect of Markov Property Violation ..... 154
11.5.2 Unbiased Sampling ..... 155
11.5.3 Spatial Reward Patterns ..... 157
11.5.4 Parallel Exploration ..... 158
11.5.5 Template Network Depth ..... 159
11.5.6 Larger Pixel Size ..... 161
11.5.7 3D FDFD Simulations ..... 161
11.5.8 Network Initialization Priors ..... 162
11.5.9 Decoupling ..... 163
11.5.10 Experimental Validation ..... 165
11.6 Conclusion ..... 167
12 Further Aspects of ONNs ..... 169
12.1 Event-driven adaptive optical neural network ..... 170
12.2 Ring Resonators as Activation Functions for Optical Neural
Networks ..... 173
12.3 Coherent dimension reduction with integrated photonic
circuits exploiting tailored disorder ..... 177
12.4 Probabilistic Photonic Computing with Chaotic Light ..... 180
12.5 Accelerating FDFD Simulations of Inverse Designed Structures
in Nanophotonics Using Deep Learning ..... 183
13 Conclusion ..... 187
Part iv Appendix
a MSAM ..... 191
a.1 Training and Implementation Details ..... 191
a.2 Details on Optimizer Comparison ..... 192
a.3 Full Hyperparameter Search Results ..... 193
b Memory Metropolis and Template Networks ..... 195
b.1 Fitness functions ..... 195
b.2 Mode Demultiplexer and wavelength Demultiplexer ..... 195
b.3 Power Splitter ..... 196
b.4 Physical Details and FDFD-simulations ..... 196
b.5 Comparison against SA ..... 197
b.6 Algorithms for Parallel Exploration ..... 198
Bibliography ..... 201