VictorJuiz/thesis huggingface dataset:
Efficient and scalable methods for deep reinforcement learning

This thesis proposes some new answers to an old question - how can artificially intelligent agents efficiently learn from their experiences to make optimal decisions? We adopt the framework of reinforcement learning (RL), in which agents are trained to maximise their expected long-term cumulative rewards, and build on the recent successes of deep RL by using deep neural network function approximators for policies, value functions, and other model components. Deep RL often learns very inefficiently from experience, and can struggle to scale to very complex problems with large action spaces or sparse feedback. We address these challenges in several ways.
In Part I, we dive deep into a subfield of RL concerning multiple agents that must cooperate to achieve a common goal. These multi-agent systems test the limits of our algorithms due to their complex dynamics, large joint action spaces, and decentralisation constraints. We develop methods to address partial observability and multi-agent credit assignment (Chapter 3), nonstationarity induced by co-learning agents (Chapter 4), and efficient representation and learning of joint action values (Chapter 5).
In Part II, we leave the specific setting of multi-agent RL to build more general inductive biases into algorithms and architectures. In Chapter 6 we leverage the inductive bias that tree-search planning is an effective representation of value functions or policies to accelerate learning. In Chapter 7 we use a curriculum of progressively growing action spaces to enable efficient exploration without compromising long-term optimality.
In Part III, we focus on estimators of higher-order derivatives in the context of RL. Among other applications, these estimators can be used in meta-learning, where we attempt to learn algorithms or inductive biases from data rather than hand-designing them as in Parts I and II. In Chapter 8 we propose an objective which may be (automatically) differentiated any number of times to produce unbiased estimates of higher-order derivatives. In Chapter 9, we extend this objective to reduce its variance, as well as allowing a flexible trade-off of bias and variance in estimators of any-order derivatives for RL.
Together, these contributions make valuable strides towards realising efficient and scalable solutions to challenging RL problems, as well as opening up exciting directions for future work building on the algorithms, architectures, and estimators proposed here.

Keywords: machine learning
multi-agent systems
reinforcement learning
deep learning
Advisor: Whiteson, S
The bureaucratic court? Administrative justice and the admissibility decision-making of the European Court of Human Rights

How should the European Court of Human Rights “decide what cases to decide”? This thesis examines the procedural justice of the admissibility decision-making of the Strasbourg court. Reappraising the “individual vs constitutional” debate which continues to dominate the Convention literature, this thesis draws on socio-legal administrative justice theory to analyse the complexities, discrepancies and tensions which have characterised debates about how Strasbourg’s docket-filtering should be organised. Having established the relevance and force of this new conceptual framework, this thesis identifies four distinct and competitive conceptions of how the Court’s docket-filtering should be designed: the Strasbourg as court, constitutional court, bureaucracy and tribunal models. I show that the ongoing debates about admissibility reform and institutional design in both the Council of Europe and the academy have been premised on neither a rigorous empirical analysis of the Court’s working practices, nor a detailed analysis of its admissibility jurisprudence.
This thesis addresses these lacunae. Drawing on an empirical analysis of the evolving text of the Convention, Council of Europe policy papers, evidence of the Court’s working practices, and a critical examination of its published admissibility decisions, this thesis demonstrates that the orthodox, doctrinal analysis of the Court’s admissibility decision-making misunderstands and misrepresents most of the Court’s working practices. The dominant voice in the literature contends that Strasbourg has now become a “constitutional court.” My data shows that while judges retain a superficial responsibility for decision-making at the threshold in Strasbourg, in practice, the Court’s Registry has quietly accrued greater and greater responsibility for the docket while judicial participation in decision-making has been marginalised. The law on the books, it is shown, remains a poor guide to how the law operates in practice. These socio-legal insights allow a radical reappraisal of our understanding of the Court, and the theoretical and methodological limits of the existing literature on the Court and the law of the Convention.

Keywords: Admissibility Decision Making
Administrative Justice
International Law
European Convention on Human Rights
Law reform
Search for Higgs Boson decays to charm quarks with the ATLAS experiment and development of novel silicon pixel detectors

The coupling of the Higgs boson to the third generation of fermions and heavy vector bosons has been experimentally well-established at the Large Hadron Collider (LHC), and the measured properties agree well with Standard Model (SM) expectation. Evidence of the Higgs coupling to second-generation quarks is still elusive, and measurements of Higgs boson decays to c-quarks are the next most promising decay mode to probe the validity of the SM.
This thesis describes a search for H → cc decays in the VH production mode, using a pp collision dataset with an integrated luminosity of 139 fb⁻¹ at a center-of-mass energy of 13 TeV, collected with the ATLAS detector. The analysis includes three different channels based on the decay of the vector boson and uses dedicated machine learning algorithms to identify c-quarks in the experiment. The observed (expected) upper limit for the VH,H → cc process is 26 (31 +12 -8) times the predicted Standard Model cross-section times branching fraction, at the 95% confidence level, and a direct constraint on the Higgs-charm coupling modifier of |κ_c| < 8.5 is set. In a combination of this analysis with a measurement of the VH,H → cc process an upper limit of |κ_c / κ_b|< 4.5 is determined.
Significant analysis improvements and a larger dataset are needed to observe H → cc decays in the future. At the High-Luminosity LHC (HL-LHC), the ATLAS detector will collect at least 3000 fb⁻¹ of integrated luminosity over ten years, which requires significant improvements to the detector to cope with increased radiation damage and higher data rates. This thesis discusses the characterisation of the ATLAS pixel detector readout chip for the inner detector upgrade (ITk) for HL-LHC. The radiation tolerance of the chip is tested in irradiations using X-rays and protons. All readout chip components, such as the digital logic and analogue front-end, are sufficiently radiation-tolerant to withstand the 1 Grad total ionising dose expected at HL-LHC.
Finally, the thesis describes monolithic pixel detectors as a possible technology for future pixel detectors and summarises the characterisation of a monolithic detector prototype in an X-ray beam test.

Advisor: Bortoletto, D
LiDAR and radar based localization with deep neural networks

Localization is a fundamental problem in many application scenarios, including robotics, computer vision and autonomous driving. In this thesis, localization is defined as two sub-tasks: odometry and relocalization. Odometry estimates the relative poses of a sensor in real-time when provided with a sequential series of sensory input data, and relocalization predicts the global poses of a sensor from previously traversed scenes. Recently, deep neural networks have been applied to visual localization and shown promising results. However, vision sensors such as cameras are greatly impacted by environmental conditions which limits the application of purely visual methods in the real world. Active sensors, such as LiDAR and radar, can capture rich geometric and semantic information in the forms of point clouds and polar scans and they are significantly more robust to environmental variations. Inspired by the recent deep visual localization methods, this thesis studies how to apply deep learning for the localization task using LiDAR and radar sensors. Specifically, this thesis studies three aspects of LiDAR and radar based localization with deep neural networks. First, an end-to-end parallel neural network termed DeepPCO is proposed for the LiDAR odometry task. It consists of two sub-networks to predict translation and orientation respectively. Second, a novel LiDAR point cloud-based method for deep LiDAR relocalization is introduced, termed PointLoc. It learns features from point clouds directly for estimating global poses in an end-to-end manner. In addition, a new indoor LiDAR-visual dataset termed vReLoc is released for studying the LiDAR relocalization problem. Third, a radar-based relocalization system, RadarLoc, is proposed by utilizing deep learning and FMCW radar scans. It designs a selfattention module with a nested encoder-decoder architecture to mask out outliers. Geometric constraints are leveraged in the loss function to improve the overall performance. Extensive experiments on large-scale datasets have been conducted to demonstrate the effectiveness of the above three approaches. This thesis also discusses implications and future research directions of this field.

Advisor: Trigoni, A
Markham, A
Fundamental physics from galaxies

Galactic-scale tests have proven to be powerful tools in constraining fundamental physics in previously under-explored regions of parameter space. In this thesis we use astrophysical systems to test some of the fundamental principles governing our current theories of the Universe, through the development of source-by-source, Monte Carlo-based forward models.
We consider modifications to the propagation of light by one of three effects: quantum gravity (QG), a non-zero photon mass and a violation of the Weak Equivalence Principle (WEP). We use spectral lag data of Gamma Ray Bursts from the BATSE satellite to constrain the photon mass to be mγ<4.0×10−5heV/c2 and the QG length scale to be ℓQG<5.3×10−18hGeV−1 at 95\% confidence, WEP to Δγ<2.1×10−15 at 1σ confidence between photon energies of 25keV and 325keV, and we demonstrate that these constraints are robust to how one models other contributions to the signal.
We investigate Galileon modified gravity theories by studying the offsets between the centre of a galaxy and its host supermassive black hole (BH). We constrain the Galileon coupling to be ΔG/GN<0.16 at 1σ confidence for Galileons with crossover scale rC≳H
−1
0
. Inspired by the aforementioned test of modified gravity, we study spatially offset BHs in the Horizon-AGN simulation and compare these to observations, finding i) the fraction of spatially offset BHs increases with cosmic time, ii) BHs live on prograde orbits in the plane of the galaxy with an orbital radius that decays with time but stalls near z=0, and iii) the magnitudes of offsets from the galaxy centres are substantially larger in the simulation than in observations.
By cross-correlating dark matter density fields inferred from the spatial distribution of galaxies with gamma ray data from the \textit{Fermi} Large Area Telescope, marginalising over uncertainties in this reconstruction, small-scale structure and parameters describing astrophysical contributions to the observed gamma ray sky, we place constraints on the dark matter annihilation cross-sections and decay rates. We rule out the thermal relic cross-section or s-wave annihilation for all mχ≲7GeV/c2 at 95\% confidence if the annihilation produces Z bosons, gluons or quarks less massive than the bottom quark. We infer a contribution to the gamma ray sky with the same spatial distribution as dark matter decay at 3.3σ. Although this could be due to dark matter decay via these channels with a decay rate Γ≈3×10−28s−1, we find that a power-law spectrum of index p=−2.75
+0.71
−0.46
is preferred by the data.
Finally, we outline a framework for assessing the reliability of the methods used in this thesis by constructing and testing more advanced models using cosmological hydrodynamical simulations. As a case study, we use the Horizon-AGN simulation to investigate warping of stellar disks and offsets between gas and stars within galaxies, which are powerful probes of screened fifth-forces.

Advisor: Ferreira, P
Potassium channel β-subunits as voltage-controlled biochemical memories

Recent work has demonstrated a role for Reactive Oxygen Species (ROS) in promoting sleep. In the Drosophila melanogaster brain, two dozen sleep-inducing neurons with projections to the dorsal fan-shaped body (dFB) integrate cumulative ROS burden over the duration of the day. The neurons’ exposure to ROS is thought to be integrated by the Kv channel Shaker’s β-subunit Hyperkinetic, whose oxidoreductase domain metabolises ROS-derived aldehydes with the help of a nicotinamide adenine dinucleotide phosphate (NADPH) co-factor. ROS exposure oxidises this co-factor to NADP+ and slows the inactivation of the Shaker-mediated A-type current, which is permissive of higher frequency neuronal activity and thereby aids sleep promotion. Typical models of homeostatic sleep posit that gradual integration of sleep promoting signals (e.g. ROS) must rapidly dissipate following sleep induction. However, the specific identity of ROS signalling molecules and how these signals are dissipated remains unclear.
Using whole cell in-vivo patch clamp recordings from dFB neurons, I demonstrate that lipid peroxidation products, such as 4-oxo-2-nonenal (4-ONE), readily interact with Hyperkinetic. 4-ONE significantly slows the A-type current inactivation and thus can act as potential physiological ROS derived messenger. Further, I show that integration of ROS molecules by Hyperkinetic can be dissipated by simulating high frequency activity within the dFB neurons. While producing a marked effect on A-type inactivation, the effects of intracellular 4-ONE in dFB neurons did not translate into an overall increased activity. Finally, assessing the sleep behaviour of a broad selection of Hyperkinetic mutants, possible amino acid residues of interest were identified as having a potential role in the long-term ROS signal integration capabilities of Hyperkinetic.

Advisor: Miesenböck, G
Waddell, S
Physics-informed machine learning: from concepts to real-world applications

Machine learning (ML) has caused a fundamental shift in how we practice science, with many now placing learning from data at the focal point of their research. As the complexity of the scientific problems we want to study increases, and the amount of data generated by today's scientific experiments grows, ML is helping to automate, accelerate and enhance traditional workflows.
Emerging at the forefront of this revolution is a field called scientific machine learning (SciML). The central goal of SciML is to more tightly combine existing scientific understanding with ML, generating powerful ML algorithms which are informed by our prior knowledge.
A plethora of approaches exist for incorporating scientific principles into ML and expectations are rising for SciML to address some of the biggest challenges in science. However, the field is burgeoning and many questions are still arising. A major one is whether SciML approaches can scale to more complex, real-world problems. Much SciML research is at a proof-of-concept stage, where techniques are validated on simplified, toy problems. Yet, understanding how well they scale to more complex problems is essential for them to become widely applicable.
This question is of central focus in this thesis. Firstly, multiple different physics-informed ML approaches are designed for three complex, real-world, domain-specific case studies taken from the fields of lunar science and geophysics, and their performance and scalability is assessed. Secondly, the scalability of physics-informed neural networks, a popular and general SciML approach, for solving differential equations with large domains and high frequency solutions is evaluated and improved. Common observations across these studies are discussed, and significant advantages and underlying limitations are identified, highlighting the importance of designing scalable SciML techniques.

Keywords: differential equations
real-world problems
domain decomposition
geophysics
scientific machine learning
physics-informed machine learning
lunar science
machine learning
scientific computing
physics-informed neural networks
deep learning
seismic simulation
permanently shadowed regions
neural networks
Advisor: Markham, A
Nissen-Meyer, T
Creativity versus conservatism in science: a simulation-based investigation


Scientists and the communities which they integrate may display stronger or weaker tendencies to conform to certain ways of approaching the research of a given scientific topic. Some have claimed that the level of conformity displayed has important epistemic implications. In particular, it has been argued that scientific nonconformity is beneficial for the progress of science, as it is conducive to the adoption of a diversity of scientific approaches, some of which may be truth-conducive and would not be attempted in more conformist settings.
In this thesis, I refine and operationalise an account introduced by Currie(2019b) which frames conformity and nonconformity in science as conservatism and creativity, respectively. In particular, I build an agent-based model from this framework, and I use it to run computer simulations and test whether scientific creativity really promotes scientific progress. Ultimately, by combining the results of these simulations with considerations about the institutions of science, I conclude that in order for a scientific community to reap the epistemic benefits often associated with scientific creativity, its scientists need not (and probably should not) be particularly creative.

Keywords: scientific creativity
nonconformity
scientific conservatism
epistemic landscape models
modelling in philosophy
conformity
agent-based model
Advisor: Koralus, P
Boundary constraints on the large-scale ocean circulation

The Meridional Overturning Circulation (MOC) is a system of surface and deep currents encompassing all ocean basins, crucial to the Earth's climate, transporting heat, carbon and nutrients around the globe. Detecting potential climatic changes in the MOC first requires a careful characterisation of its inherent variability. Key components of the MOC are the Atlantic MOC (AMOC) and the Antarctic Circumpolar Current (ACC).
In this thesis, the role of boundary properties (including density and velocity) in determining the AMOC and ACC is investigated, as a function of cross-sectional coordinate and depth, using a hierarchy of general circulation models. The AMOC is decomposed as the sum of near-surface Ekman, depth-independent bottom velocity and eastern and western boundary density components. The decomposition proves a useful low-dimensional characterisation of the full 3-D overturning circulation, providing a means to quantify the relative importance of boundary contributions on different timescales.
The estimated total basin-wide AMOC overturning streamfunction, reconstructed using only boundary information, is in good agreement with direct calculations of the overturning using meridional velocities, in terms of magnitude and spatial structure. The time-mean maximum overturning streamfunction is relatively constant with latitude, despite its underlying boundary contributions varying considerably, especially in the northern hemisphere. Regression modelling, supplemented by Correlation Adjusted coRrelation (CAR) score diagnostics, provides a natural ranking of the various boundary components in explaining total transport variability. At short timescales the bottom, western boundary and Ekman components play a dominant role. At decadal timescales, the importance of both boundary density components is revealed.
Applying a similar decomposition diagnostic to the ACC provides insight into the differences between model simulations, revealing spatial resolution dependence of ACC transport through the Drake Passage. All models exhibit a weak ACC compared to observations, the 1/4∘ model showing significant weakening over the first 40 years of simulation. The weaker ACC transport in the 1/4∘ model can be attributed to fresher, lighter southern boundary densities and a strengthened Antarctic Slope Current.
Density maps along sloping ocean boundaries are produced, incorporating the western and eastern Atlantic boundaries and the Antarctic coastline. Isopycnals are flat over long portions of eastern ocean boundaries, and slope linearly on western boundaries; possible mechanisms for these features are discussed. The relationship between boundary currents and isopycnal slope around Antarctica is also considered.
Results of this thesis serve to indicate the importance of boundary information in characterising the AMOC and the ACC, and the relative simplicity of along-sloping-boundary density structure.

Keywords: variability
boundary properties
CAR scores
ACC
decomposition
AMOC
MOC
Advisor: Johnson, H
Marshall, D
Bell, M
Engineering terpenoid oxidation by cytochrome P450BM3 for fine chemical synthesis

P450BM3 (CYP102A1) has a unique structure of having the electron-transfer reductase domain fused to the haem monooxygenase domain. The intramolecular electron transfer process makes P450BM3 catalytically self-sufficient and promotes high turnover rates. P450 enzymes activate C−H bonds via the insertion of an oxygen atom from atmospheric dioxygen and play key roles in the biosynthesis of endogenous compounds (terpenes, alkaloids and steroids).
Oxygenated terpenoid compounds have desirable properties such as aromas, flavours and medicinal effects. The initial oxygenation step in terpenoid biosynthesis from the terpene precursors is invariably catalysed by a P450 enzyme. Chapter 3 describes the engineering and application of P450BM3 variants to develop efficient, two-step chemo-enzymatic synthesis routes to high value the norisoprenoid and isoprenoid aroma compounds b-damascenone, tabanone isomers, and rose oxide by late-stage C−H activation.
The generation of diverse metabolites of common steroids via P450-catalysed direct C−H activation has important applications in the synthesis of steroidal agents for drug screening and development. Chapter 4 describes the engineering of P450BM3 for the mono- and di-hydroxylation of androstenedione (AD), dehydroepiandrosterone (DHEA) and testosterone (TST). To design altered steroid binding orientations to broaden the regioselectivity, the structures of wild type P450BM3 and the steroid aromatase CYP19A1 are overlaid to identify key regions for scanning glycine mutagenesis which greatly expanded the product range of AD, DHEA and TST. In Chapter 5, more steroidal compounds are screened to show hydroxylation at multiple sites with high regio- and stereo-selectivity for androsterone (AN), epi-androsterone (EAN), progesterone (PROG) and pregnenolone (PREG).

Keywords: steroids
synthesis
norisoprenoid
P450
terpenoid
oxidation
Advisor: Wong, L