VictorJuiz/thesis huggingface dataset:
Reinforcement learning in large state action spaces
Keywords: multi agent systems
variational methods
high dimensional systems
generalization in AI
sequential decision making
state abstraction
reinforcement learning

Reinforcement learning (RL) is a promising framework for training intelligent agents which learn to optimize long term utility by directly interacting with the environment. Creating RL methods which scale to large state-action spaces is a critical problem towards ensuring real world deployment of RL systems. However, several challenges limit the applicability of RL to large scale settings. These include difficulties with exploration, low sample efficiency, computational intractability, task constraints like decentralization and lack of guarantees about important properties like performance, generalization and robustness in potentially unseen scenarios.
This thesis is motivated towards bridging the aforementioned gap. We propose several principled algorithms and frameworks for studying and addressing the above challenges RL. The proposed methods cover a wide range of RL settings (single and multi-agent systems (MAS) with all the variations in the latter, prediction and control, model-based and model-free methods, value-based and policy-based methods). In this work we propose the first results on several different problems: e.g. tensorization of the Bellman equation which allows exponential sample efficiency gains (Chapter 4), provable suboptimality arising from structural constraints in MAS(Chapter 3), combinatorial generalization results in cooperative MAS(Chapter 5), generalization results on observation shifts(Chapter 7), learning deterministic policies in a probabilistic RL framework(Chapter 6). Our algorithms exhibit provably enhanced performance and sample efficiency along with better scalability. Additionally, we also shed light on generalization aspects of the agents under different frameworks. These properties have been been driven by the use of several advanced tools (e.g. statistical machine learning, state abstraction, variational inference, tensor theory).
In summary, the contributions in this thesis significantly advance progress towards making RL agents ready for large scale, real world applications.
Investigating triggering events in T and B lymphocytes
Keywords: microscopy
immunology
immune cell signalling

T cells have an important role in the resolution of infections and cancer. The mechanisms underlying early contact formation of T cells with antigen-presenting cells, and the contribution of T-cell receptor (TCR) microclusters (MCs) to T-cell activation, are still not well understood, especially given recent evidence demonstrating the importance of membrane topography in T-cell signalling. Furthermore, it is unclear what features of the TCR triggering mechanism are shared with other receptors, such as the B-cell receptor (BCR). For experiments presented in this thesis, cell lines were created that express TCRs with different affinities and surface densities, along with an inhibitable mutant of ZAP-70, which were used to probe the effects of TCR density and signalling in early T-cell contacts and antigen reactivity. TCR density did not affect the spreading or close contact formation of T cells with supported lipid bilayers (SLB) presenting agonist pMHC, suggesting that very few TCRs are needed to trigger these events. MC distribution was uncorrelated with close contacts, implying that these structures are real, and not imaging artefacts. However, T cells with low TCR expression levels signalled robustly, suggesting that MC formation might not be a pre-requisite of TCR signalling. Nevertheless, signalling triggered under these conditions did not result in full T-cell activation. ZAP-70 inhibited cells displayed reduced cell spreading, formed fewer but larger close contacts, and recruited more ZAP-70 at MCs. This data suggests that ZAP-70 is a strong regulator of T-cell contact formation, and that it also contributes to antigen discrimination, alongside its key role in signal amplification at the TCR. Finally, key requirements of the kinetic segregation (KS) model were confirmed in T and B cells, including the induction of TCR triggering by slowing the receptor’s diffusion within a close contact, and the demonstration that the BCR is unclustered on resting cells, similarly to the TCR, thereby ruling out a popular model of BCR triggering. Work carried out in this thesis provides insight into the mechanisms regulating early contact formation by T cells, and by demonstrating key predictions of the KS model in both T and B cells, raises the possibility that a unified mechanism for immune receptor triggering could exist.
Modelling medulloblastoma using cerebellar organoids
Keywords: CRISPR
induced pluripotent stem cells
cerebellum
disease modelling
organoids
Sonic Hedgehog
medulloblastoma

Medulloblastoma (MB) is the most common malignant brain tumour in children and originates from cerebellar progenitor cells. Although extensive genomic studies have resulted in detailed molecular signatures of MB subtypes, current preclinical models often fail to capture human tumour biology accurately. Furthermore, available in vitro models do not harbour the most frequent mutations, limiting the translational potential of preclinical findings.
In this thesis, I describe the use of induced pluripotent stem cells (iPSC) and cerebellar organoid differentiation for MB modelling. Using CRISPR/Cas9 gene editing, I introduce a loss-of-function (LOF) mutation in the PTCH1 gene, the most frequently mutated gene in the Sonic hedgehog (SHH) type of MB. This subtype is characterised by increased signalling of the SHH signalling pathway. I show that homozygous LOF results in changes in differentiation trajectory, while heterozygous mutants differentiate into cerebellar organoids and display features related to preneoplasia. In addition, I describe potent alternative strategies for MB modelling using the introduction of driver genes in differentiated cerebellar organoids that results in proliferative growths and tumorigenesis when transplanted in mice. Cerebellar organoid co-culture with established MB cell lines provides a third strategy and enables the study of the tumour microenvironment.
Together, these models capture various aspects of tumorigenesis and allow the study of human MB biology. The application of these strategies may overcome the current limitations of preclinical models and help bridge the gap from bench to bedside.
Development of novel quantitative methods for monitoring neurodegeneratie phenotypes in Niemann-Pick disease type C mice
Keywords: gait
catwalk
NPC
Niemann-Pick disease Type C
deeplabcut

A hallmark of neurodegenerative diseases, such as the rare monogenic lysosomal storage disease, Niemann-Pick disease type C (NPC), is motor dysfunction. There- fore, gait analysis is frequently used in pre-clinical and clinical studies. The Catwalk system is an automated instrument that collects gait data from rodents. The system captures data on 495 variables. However, there is currently no standard process to identify disease relevant variables or indeed analyse these variables. Therefore, researchers report only a subset of Catwalk data based on assumptions, hypotheses, or the level of statistical significance. Thus, highly relevant variables might be overlooked, and this might lead to misleading or ill-informed conclusions to be drawn. In addition, any disease model that results in reduced body weight (e.g., Npc1 -/- mice) results in weak paw print signals in the Catwalk system, which will affect the quality and quantity of the data. In this thesis, I have therefore innovated a new standard process, the Evidence Based Gait Analysis Method, that not only optimises the quality and the quantity of the data but also provides a systematic unbiased basis for selecting relevant variables. Cerebellar ataxia is a key feature in Npc1 -/- mice and results from the loss of Purkinje cells in the cerebellum. The Catwalk system can detect this trait as it generates paw-oriented variables. However, these variables are insufficient to describe the whole picture of the ataxia these mice display, particularly the angle of deviation from movement in a straight trajectory. Thus, an additional walking path method was developed to complement the gait analysis method. The walking path was tracked using DeepLabCut (DLC), a deep learning algorithm, using videos recorded by the Catwalk system. The paths were analysed in four aspects, namely velocity, X-direction, Y-direction, and angular rotation. This can capture the walking phenotype of mice and the interpretation of the result is therefore straightforward. Together, these two new methods offer for the first time a detailed evaluation of gait performance underpinned by robust statistical evidence and so is a potential game changer in this field. I have applied this new and innovative approach to analysing changes in gait in two mouse models of NPC disease either untreated or treated with single or combination therapies and the data generated allow us for the first time to study in detail the differential responses of the disease to different treatments. We anticipate that this will form the basis for the rational selection of optimal combination therapies for clinical application in the future.
Using somatic evolution to unravel cell fate dynamics to gain tissue level insights
Keywords: machine learning
epidermis
cancer initiation
evolution
somatic evolution
cell fate dynamics
homeostasis
stem cells
mathematical model
tumorigenesis
hematopoiesis
clonal evolution

Cancer initiation and progression is an evolutionary process where somatic alterations convey fitness advantages to cells that exist in a complex environment acting to shape cell populations. Homeostasis, being the normal state of a tissue prior to disease, prioritizes and protects long lived stem cells that are capable of incurring these somatic alterations and contributes to the early evolutionary dynamics from which cancer initiation occurs. Research over the past two decades has made substantial inroads into our understanding of somatic mutations through characterization of healthy tissue. In parallel, agent based mechanistic models have emerged as an important tool for understanding somatic mutation and stem cell dynamics in both cancer and homeostatic tissues. In this thesis we combine first principles mathematical modeling of homeostasis, complete with high-resolution genotypic information, with healthy patient sequence data in various tissue types to answer critical biological questions about somatic evolution, stem cell dynamics, and cancer initiation.
We first develop a three-dimensional mechanistic model of the homeostatic epidermis and methodology necessary to incorporate base pair resolution mutation data. We use this model to examine neutrally expanding subclonal populations and compare simulations to healthy patient data. We then introduce functionally heterogenous oncogenic drivers in our in-silico model to examine non-neutral dynamics in the epidermis. Next, we develop a novel lineage tracing method using fluctuating methylation sites through analysis of patient data combined with a spatial model of a homeostatic intestinal crypt to determine stem cell dynamics in glandular tissue. Our spatial model reveals the necessary dynamics of fluctuating methylation sites to be used as fluctuating methylation clocks that allows us to determine stem cell behavior over short timescales in vivo.
In the last chapters of this thesis, we focus on using our novel fluctuating methylation clock and somatic evolution as an important biomarker to understand clonal hematopoiesis. As the hematopoietic stem cells become homogenized through disease, a distinctive distribution emerges within patient data that can be exploited to diagnose clonal hematopoiesis without the need to understand somatic drivers. Combined, the work in this thesis reveals the power of somatic evolution on various timescales as a marker that can provide tissue level insights. We conclude by outlining a future direction where combining fluctuating methylation markers with DNA alteration could be a powerful tool in diagnostics and prognosis within the clinic.
Biobank-scale ancestral recombination graphs: inference and applications to the analysis of complex traits
Keywords: statistical genetics
ancestral recombination graph
population genetics

Across living species, DNA is transmitted from generation to generation via the processes of inheritance, mutation, and recombination. The history of these processes can be recorded using genome-wide gene genealogies. Accurate inference of gene genealogies from genetic data has the potential to facilitate a wide range of analyses, but is computationally challenging. In this thesis, we introduce a scalable method, called ARG-Needle, that uses genotype hashing and a coalescent hidden Markov model to infer genome-wide genealogies from sequencing or genotyping array data in modern biobanks. We develop strategies that utilise the inferred genome-wide genealogies within linear mixed models to perform association and other analyses of biomedical traits.
We validate the accuracy and scalability of ARG-Needle through extensive coalescent simulations, and use ARG-Needle to build genome-wide genealogies from genotypes of 337,464 UK Biobank individuals. We perform genealogy-based association analysis of 7 complex traits, detecting more rare and ultra-rare signals (N = 133, frequency range 0.0004% − 0.1%) than genotype imputation from ∼65,000 sequenced haplotypes (N = 65). We validate these signals using exome sequencing data from 138,039 individuals. ARG-Needle associations strongly tag (average r = 0.72) underlying sequencing variants that are enriched for missense (2.3×) and loss-of-function (4.5×) variation. Compared to imputation, inferred genealogies also capture additional signals for higher frequency variants. These results demonstrate that biobank-scale inference of gene genealogies may be leveraged in the analysis of complex traits, complementing approaches that require the availability of large, population-specific sequencing panels.
A minimalist approach to deep multi-task learning
Keywords: reinforcement learning
deep learning
GNN
multi-task learning
machine learning
transformers
graph neural networks

Multi-task learning is critical for real-life applications of machine learning. Modern approaches are characterised by algorithmic complexity, often unjustified, leading to impractical solutions. In contrast, this thesis demonstrates that a minimalistic alternative is possible, showing the attractiveness of simple methods. 'In defence of the Unitary Scalarisation for Deep Multi-task Learning' motivates the rest of the thesis, showing that none of the more complex multi-task optimisers outperforms the simple per-task gradient summation when compared on fair grounds. Furthermore, it proposes a novel look at multi-task optimisers from the regularisation standpoint. The rest of this thesis focuses on deep reinforcement learning, a general framework for sequential decision-making. In particular, we look at the setting when observations (inputs to the model) are represented as graphs, i.e., collections of interconnected nodes. In 'Scaling GNNs to High-Dimensional Continuous Control' and 'The Role of Morphology in Graph-Based Incompatible Control', we learn a single control policy for agents of different morphology by representing the observation set elements as graphs and deploy graph neural networks (including transformers). In the former chapter, we devise a simple method to scale graph networks by freezing some parts of the network to stabilise learning and prevent overfitting. In the latter chapter, we show that graph connectivity might be suboptimal for the downstream task demonstrating that less-constrained transformers perform significantly better without having the graph connectivity information. Finally, in the 'Generalisable Branching Heuristic for a SAT Solver', we apply multi-task reinforcement learning to Boolean satisfiability, a fundamental problem in academia and industrial applications. We demonstrate that Q-learning, a staple reinforcement learning algorithm equipped with graph neural networks for function approximation, can learn a generalisable branching heuristic.
We hope our findings will steer the further development of the field: creating more complex benchmarks, adding assumptions on task similarities and a model capacity, and exploring other objective functions rather than focusing on the average performance across the tasks.
From visuomotor control to latent space planning for robot manipulation
Keywords: visuomotor control
latent space planning
robot manipulation

Deep visuomotor control is emerging as an active research area for robot manipulation. Recent advances in learning sensory and motor systems in an end-to-end manner have achieved remarkable performance across a range of complex tasks. Nevertheless, a few limitations restrict visuomotor control from being more widely adopted as the de facto choice when facing a manipulation task on a real robotic platform. First, imitation learning-based visuomotor control approaches tend to suffer from the inability to recover from an out-of-distribution state caused by compounding errors. Second, the lack of versatility in task definition limits skill generalisability. Finally, the training data acquisition process and domain transfer are often impractical. In this thesis, individual solutions are proposed to address each of these issues.
In the first part, we find policy uncertainty to be an effective indicator of potential failure cases, in which the robot is stuck in out-of-distribution states. On this basis, we introduce a novel uncertainty-based approach to detect potential failure cases and a recovery strategy based on action-conditioned uncertainty predictions. Then, we propose to employ visual dynamics approximation to our model architecture to capture the motion of the robot arm instead of the static scene background, making it possible to learn versatile skill primitives. In the second part, taking inspiration from the recent progress in latent space planning, we propose a gradient-based optimisation method operating within the latent space of a deep generative model for motion planning. Our approach bypasses the traditional computational challenges encountered by established planning algorithms, and has the capability to specify novel constraints easily and handle multiple constraints simultaneously. Moreover, the training data comes from simple random motor-babbling of kinematically feasible robot states. Our real-world experiments further illustrate that our latent space planning approach can handle both open and closed-loop planning in challenging environments such as heavily cluttered or dynamic scenes. This leads to the first, to our knowledge, closed-loop motion planning algorithm that can incorporate novel custom constraints, and lays the foundation for more complex manipulation tasks.
Cognitive impairment following stroke: prevalence, predictors, and trajectories of domain-specific deficits
Keywords: stroke
cognitive impairment

Cognition is not a unitary construct and encompasses multiple domains, therefore it is critical to consider the diverse range of deficits a stroke survivor may experience and how different impairments might impact other stroke outcomes. The purpose of this thesis is to investigate the prevalence, trajectories, impact, risk factors, and measures of both general and domain-specific cognitive impairment after stroke. First, I determined the prevalence and predictive value of domain-specific cognitive impairments measured with the Oxford Cognitive Screen in a cohort of 866 acute stroke patients, 430 of which completed 6-month follow-up. Next, I investigated the prevalence of acute infection, positive systemic inflammatory response syndrome (SIRS), and elevated inflammatory markers at and during hospital admission, and examined the associations of acute inflammatory markers and systemic infection, with cognitive function in a subset of 255 stroke patients. Subsequently, I analyzed the relationship between domain-general and domain-specific cognitive impairment and overall quality of life 6 months after stroke, as measured by the Stroke Impact Scale. Next, I completed a systematic review and meta-analysis investigating the prevalence of cognitive impairment beyond the first year after stroke and the methods being used to assess cognition in long-term survivors. Lastly, I examined the prevalence of domain-specific cognitive impairment in a chronic stroke cohort of at least 2 years post-stroke, determined the relationship between acute, 6-month and long-term cognitive function, as well as assessed longitudinal trajectories of domain-specific cognitive changes. The findings of these five empirical chapters are then discussed collectively to clarify the prevalence and temporal evolution of domain-specific impairments after stroke, expand the current knowledge of how cognitive impairment may impact functional and mood outcomes after stroke, and establish whether acute domain impairments and systemic inflammation may serve as risk factors for longer-term cognitive impairment. Finally, I highlight potential clinical and research implications, as well as future directions.
Symmetric instability in the Atlantic meridional overturning circulation
Keywords: mixing
geophysical fluid dynamics
climate
baroclinic instability
barotropic instability
AMOC
submesoscale instability
Atlantic
Atlantic Meridional Overturning Circulation
symmetric instability
ocean

The role of symmetric instability in mixing momentum, buoyancy and poten- tial vorticity is examined in several constituent currents of the Atlantic Meridional Overturning Circulation: the North Brazil Current and its rings, the Deep Western Boundary Current in the Tropical Atlantic, and the East Greenland Current of the Sub-polar North Atlantic.
Linear stability theory predicts the structure and growth rates of overturning cells generated by symmetric instability in cross-equatorial western boundary cur- rent systems. Stratification, mixing, and curvature in the velocity field stabilise these symmetrically unstable motions. A measure of the importance of the non- traditional component of the Coriolis force in modifying symmetrically unstable motions is derived.
Idealised models of the North Brazil Current system are used to assess predic- tions from the linear stability analysis. They show symmetric instability is effective at neutralising negative potential vorticity originating from the Southern Hemi- sphere, enabling large-scale interhemispheric water exchange. Symmetric insta- bility drives ∼ 4 × 106 m3 s−1 of water mass transformation between intermediate and deep waters in this system.
Idealised models of the Deep Western Boundary Current demonstrate that sym- metric instability remains an efficient mechanism for mixing potential vorticity despite dynamical differences to the North Brazil Current. Overturning cells pro- duced by the instability generate density staircases. These provide a new diagnos- tic for the detection of symmetric instability.
Finally, an ensemble of idealised models of the East Greenland Current are forced with varying down front winds, which generate Ekman buoyancy fluxes. These fluxes deepen the convectively mixed layer and through the excitement of symmetric instability create a deeper low potential vorticity layer below it. This layer has potential vorticity of approximately zero and near zero stratification.
Symmetric instability is an efficient mechanism for mixing momentum, buoy- ancy and potential vorticity in both tropical and high-latitude components of the Atlantic’s overturning circulation.