VictorJuiz/thesis huggingface dataset:
Reward Machines

Reinforcement learning involves the study of how to solve sequential decision-making problems using minimal supervision or prior knowledge. In contrast to most methods for automated decision making, reinforcement learning agents do not require access to a formal description of the problem in order to find optimal solutions. Instead, such agents learn optimal behaviour using a trial-and-error strategy. This learning strategy makes reinforcement learning a strong candidate to tackle real-world problems with complex dynamics. However, there are two core challenges that limit the use of reinforcement learning in the real world: sample efficiency and partial observability. In this dissertation, we tackle these two problems through the creation of reward machines. Reward machines are automata-based representations of a reward function that expose reward structures to the agent. We show that agents can exploit these structures to improve their sample efficiency and their performance under partial observability. In particular, we propose (i) to use decomposition methods and reward shaping to improve sample efficiency based on a given reward machine, and (ii) to use reward machines as external memory and solve partially observable tasks by learning reward machines. In both cases, we provide theoretical and empirical evidence of the benefits of utilizing reward machines to tackle these problems. Finally, we conclude this dissertation with a discussion of the role that reward machines can play in tackling other long-standing problems in reinforcement learning, such as developing agents with interpretable behaviours, providing guarantees that an optimal policy is safe, and creating agents that can understand instructions.

Keywords: Artificial Intelligence
Knowledge Representation
Partial Observability
Reinforcement Learning
Reward Machines
Reward Specification
Advisor: McIlraith, Sheila Ann
Essays in Economic Growth

This thesis presents three studies in economic growth. The first chapter (with Marina Tavares) examines a novel channel through which financial shocks are transmitted across countries, the international capital markets of multinationals. We exploit empirical variation in the availability of credit to U.S. multinationals during the 2008-09 financial crisis to identify the strength of this channel. We estimate that the U.S. credit crunch accounted for a quarter of the drop in sales of European subsidiaries during the crisis. Through the lens of a dynamic multicountry model, our micro estimates imply that financial linkages within multinationals account for at most a fifth of global GDP comovement. Our findings suggest that countries with the largest foreign multinational presence experience more spillovers from foreign financial shocks, especially from shocks originating in the United States. The second chapter proposes a tractable quantitative framework to examine the role of inter-industry productivity spillovers in the catch-up growth of open economies. First, I document that a country’s comparative advantage tends to increase in industries that employ occupations that are used most intensively in current exports. The model rationalizes these patterns by incorporating occupation-specific dynamic scale economies into a multi-sector gravity framework. I estimate that scale economies are relatively large in high-skilled production. Simulations suggest that spillovers play a quantitatively substantial role in accounting for slow cross-country convergence and increase the gains from trade, especially in countries with a comparative advantage in manufacturing. The third chapter (with Swapnika Rachapalli and Diego Restuccia) uses variation in land-market institutions across Indian states and detailed micro panel data to study distortions in land rental markets and their impact on agricultural productivity. We find empirical evidence that states with more rental-market activity feature less misallocation and over time reallocate land more efficiently. We develop a model of land rentals across heterogeneous farms to estimate land-market distortions in each state and assess their quantitative effect on agricultural productivity. Land rentals have substantial positive effects on agricultural productivity: an efficient reallocation of land increases agricultural productivity by 38 percent on average and by more than 50 percent in states with highly distorted rental markets.

Advisor: Restuccia, Diego
Young Black Men's Embodied Experiences of Traumatic Gun Violence

This thesis examines the experiences of 10 young Black men who have been impacted by traumatic gun violence either directly through being shot or indirectly by witnessing gun violence in their communities or losing loved ones to gun violence. Few studies have considered the traumatic and embodied impact of gun violence on young Black men. Using trauma theory and embodiment as theoretical frameworks, this research demonstrates how due to the lack of resources and recognition of young Black men’s experiences of traumatic gun violence, young Black men begin to “embody” experiences of traumatic gun violence physically and emotionally. Young Black men’s experiences of gun violence within the city of Toronto continue to be stigmatized, criminalized, and disregarded as a traumatic issue and the main approaches to solving gun violence have focused on criminal justice interventions and policing which do not engage with the actual “embodied trauma” of gun violence. This thesis demonstrates that to understand how young Black men embody traumatic violence we first need to understand how systemic violence and structural oppression, as expressed through racial profiling, over-surveillance, incarceration, police brutality, and experiences of poverty, coincide with the traumatic impact of gun violence, and directly relate to young Black men’s experiences of ongoing exposure to gun violence.

Keywords: Black male youth
Embodiment
Gun Violence
Masculinity
Systemic Violence
Trauma
Advisor: McCready, Lance T
Interorganizational Relationships in British Columbia’s Community Overdose Response: Evaluating Community Partnerships as a Network Intervention

Background: Surging overdose deaths in British Columbia (B.C.) prompted the development of Community Action Teams (CATs) - interorganizational networks that deliver overdose prevention strategies. The formation of CATs is a network intervention, aiming to optimize implementation by fostering connectivity. My research questions were: 1) How were the network intervention components (relational antecedents, structure, processes and outcomes) enacted in each of the participating CATs, and how (and why) were these similar or different?; 2) How did each of the CAT components influence strategic goals/collaborative objectives? Methods: This was a multiple case study using multiple methods. I adapted a framework for evaluating interorganizational relationships (IORs ) to understand partnership building relative to implementation goals/tasks. I selected 5 CATs as cases and used multiple sources of data to compile the case studies: social network survey; community survey; interviews; and document review. I conducted a social network analysis, a descriptive analysis of the community survey, a thematic analysis of interview transcripts, and a content analysis of key themes from documents. Findings were compiled into summary tables and narratively integrated. Results: Three cases were ultimately included. CAT 1 took on a ‘whole network’ approach in which strategic goals involved most CAT members and pertained to enhancing partnering. Structures, processes and outcomes in this CAT were more participatory and cohesive. CAT 2 funded small projects, which led to a few organizations participating in pockets of action. This may have led to lower network cohesiveness, higher centralization (compared to CAT 1), and relational challenges. CAT 3 funded small projects involving the participation of many CAT members, which fostered better relational processes compared to CAT 2. Formality and fairness were key convening processes. All three CATs were operating network-wide at a mid-range level of collaboration, and all met their implementation tasks but not necessarily their collaboration goals. Conclusion: Information about implementation tasks/goals can help us better understand network formation. Evaluating IORs as network interventions can lend insights about how and why certain networks are built in certain ways, and better enables us to study and provide guidance on these relationships in the future.

Keywords: community collaboration
networks
overdose prevention
Advisor: Berat, Whitney B
Applying Precision Genome Editing Technologies to Variant Effect Interpretation

Over the last two decades, advances in sequencing technologies have enabled routine implementation of genome sequencing in clinical practice. The utility of this sequencing data, however, is undermined by a poor understanding of the functional or clinical consequences of genetics variants found in human genomes. The realization of individualized or precision medicine will require a narrowing of the gap between our ability to read genomes and our ability to interpret their contents. This is being facilitated by the emergence and application of genome editing technologies – particularly those based on the CRISPR-Cas system. These approaches not only allow for the deliberate introduction of virtually any mutation-of-interest into a genome, but also enable complex genome engineering required for the establishment of tractable or relevant model systems. The most recent iteration of CRISPR-Cas development has led to a variety of ‘precision genome editors’ – namely base and prime editors – which permit facile nucleotide editing in a targeted, predictable and highly efficient manner. Here, I describe the implementation of precision genome editors to variant effect interpretation in the pediatric lysosomal storage disorder Niemann-Pick disease type C (NPC). First, I demonstrated the utility of combining base editors with a haploid cell model to rapidly establish models of NPC for molecular characterization. Next, I applied multiplexed prime editing to enable high-throughput variant interpretation in a strategy I call “saturation prime editing”. This was enabled by a methodology I developed based on allele-specific genome editing that allows for targeted “haploidization” of gene loci. Having only a single allele present in a cell – and thus a single allele driving an observed phenotype – allows for accurate mapping of genotype-to-phenotype using next-generation sequencing. By combining saturation prime editing with cell line haploidization, I derived function scores and clinical interpretations for more than 900 different variants in the NPC1 gene underlying NPC. I anticipate the broad application of these strategies will further our understanding of genetic variation and increase the clinical yield of diagnostic or preventative sequencing efforts.

Advisor: Cohn, Ronald D
Deeply Conserved Cardiac Enhancers Guide Zebrafish Heart Regeneration

Adult mammalian hearts do not replenish lost cardiomyocytes (CMs) after injury, replacing them with akinetic scar tissue. One strategy for promoting cardiac healing in humans involves re-activation of regeneration programs conserved through evolutionary history in vertebrates such as the teleost fish Danio rerio, commonly known as the zebrafish. The zebrafish heart robustly regenerates cardiac tissue in multiple injury models, restoring myocardium via pre-existing CMs that de-differentiate assume a proliferative cellular phenotype. Non-coding cis-regulatory regions such as enhancers determine the spatiotemporal dynamics of gene expression and have been leveraged as tools for deployment of pro-regenerative factors into the hearts of non-regenerative species. Despite prediction of developmentally linked enhancers for promoting adult cardiac regeneration, there has been a dearth of such enhancers reported in the literature. In this thesis, I examine the role of conserved cardiac enhancers identified as active in zebrafish embryos during the phylotypic period of development. These enhancers represent Accessible Conserved Non-coding Elements (aCNEs) that label specific cardiac compartments during cardiac development and regeneration, with several active in wound-proximal endocardium and de-differentiated myocardium. I find evidence that orthologous human aCNEs fail to activate in response to cardiac injury, implicating divergent evolutionary trajectories for post-injury enhancer activation between species. I also utilize the regulatory sequence representing an enhancer for hey2 (aCNE 21), de-coupled from endogenous hey2 expression in the adult heart, to define the contributions of trabecular myocardium to zebrafish heart regeneration. Through mutational analysis of a hand2-linked enhancer element (aCNE 1), I uncover critical activating and repressive regulatory modules required for regulation of enhancer activity during development and regeneration. Finally, genetic deletion of aCNE 1 reveals an essential role for directing the cardiac repair process. Loss-of-enhancer mutants retain scar tissue and fail to regenerate; aCNE 1 mutants further display impaired recruitment of leukocytes and dysregulation of matrix metalloproteinase activity within the wound area. Together, these results serve to identify regulatory elements with the unique ability to target gene expression to specific cardiac compartments, and further explain the differences in cardiac regenerative potential between regenerative and non-regenerative species.

Keywords: Cardiac
Development
Enhancers
Heart
Regeneration
Zebrafish
Advisor: Scott, Ian C
Developmental Stage-dependent Effects of Embryonic Ethanol Exposure on Social Behaviour and Potential Underlying Mechanisms in Adult Zebrafish: A Model for FASD

Fetal Alcohol Spectrum Disorder (FASD) is characterized by a variety of morphological, behavioural, and cognitive deficits, including life-long learning disabilities and social dysfunction. FASD is a highly variable disorder, and symptoms range from mild to severe despite similar drinking patterns in mothers. One of the hypotheses proposed to explain this variability is the timing of alcohol consumption, or in other words, the developmental stage at which the embryo is exposed to alcohol. Due to their highly social nature, the zebrafish is becoming a popular model organism for investigating social deficits that arise from embryonic ethanol exposure. Previous studies in zebrafish have observed ethanol administered at 24 hours post fertilization (hpf) resulted in abnormal social behaviour accompanied by lower levels of dopamine and serotonin in the brain, however, little is known about how ethanol alters social behaviour and neurochemical systems at other developmental timepoints, and the mechanisms underlying these alterations remain unknown. Therefore, the goal of this thesis was two-fold: firstly, to investigate how the developmental stage of embryonic ethanol exposure differentially alters social behaviour and underlying neurochemistry, and second, to investigate novel mechanisms underlying embryonic ethanol-induced social deficits in adult zebrafish. Results presented here show both social behaviour and underlying neurochemistry are indeed impaired by ethanol in a developmental stage-dependent manner, however, the expression of ethanol-induced neurochemical dysfunction differed slightly depending on the age of observation. In adult zebrafish, abnormal social behaviour was accompanied by lower levels of whole brain dopamine, specifically when ethanol was administered at 10hpf, 16hpf and 24hpf, however, ethanol administered earlier (5hpf) and later (36hpf) had little to no effect. Additionally, abnormal social behaviour resulting from ethanol exposure at 24hpf was accompanied by lower levels of whole brain isotocin in adult zebrafish. Lastly, exposure to ethanol at 24hpf resulted in altered histone variant and neurochemical gene expression in adult zebrafish. These findings not only point to sensitive periods during early development but highlight potential new molecular candidates which may play an important role in the expression of embryonic ethanol-induced social deficits.

Keywords: Dopamine
Fetal Alcohol Spectrum Disorder
Gene expression
Isotocin
Social behaviour
Zebrafish
Advisor: Gerlai, Robert T
An Exploratory Study of Kindergarten Children’s Understandings About the Pedagogical and Social Nature of Kindergarten

This study was conducted to investigate what children saw as the pedagogical and social purposes of their kindergarten experiences during the 2010-2011 school year when education in Ontario began a transition in the length and focus of kindergarten programs. In a large suburban region of Ontario, junior kindergarten (JK) children and senior kindergarten (SK) children (N=263) from kindergarten programs in 12 elementary schools were interviewed, with sampling from the new Full-Day Early Learning Kindergarten Program (FDELK) as well as from traditional kindergarten programs where it had not yet been implemented. Using a puppet technique, children were interviewed about the pedagogical aspects of kindergarten, such as what activities they liked and what they felt were important, and what their teachers did. They were also asked about the social nature of kindergarten, for example how they would solve social problems in kindergarten, including dealing with social conflicts involving themselves and other children. Children’s responses to interview questions demonstrated that they distinguished between ‘play’ and ‘work’ (academic skills such as pre-reading and numeracy), in the activities they favoured (play) and in those they saw as being important (work), and in what they perceived their teachers do (work). The study showed a relationship between children’s responses to the interview questions and higher receptive vocabulary and early reading ability scores. Children who mentioned work-related concepts and who reported multiple strategies in peer conflict situations had higher scores in vocabulary and early reading. A common theme across most children’s responses was the importance that they attached to learning rules and to following them. Children associated their teachers’ roles with behaviour guidance and rule enforcement in addition to leading instruction and distributing academic work. Few children mentioned that their teachers’ role was to play with them. There were slight demographic differences found in this study. SK children’s pedagogical perceptions were work- and rule-related, and social perceptions included more strategic solutions to conflict in comparison to JK children. English language learners (ELL) were more likely to mention play and work to be important, and their teachers’ role to be work-related compared to English first language children (EFL).

Keywords: children's voices
full day learning
kindergarten
perception
play
social development
Advisor: Pelletier, Janette
Quantum Parameter Estimation in the Laboratory

In this thesis, I report on three projects which I have completed during my PhD. All of these projects are related to quantum parameter estimation.When simultaneously measuring all the parameters describing a unitary transformation, a decision needs to be made regarding the relative importance of these different parameters. In our first project, we show that the geometry of the group of unitaries in question can guide us toward a natural way of weighting these dif- ferent parameters. This allows us to transform the matrix Cram ́er-Rao inequality into a simple parameter-independent scalar inequality. We use this new inequality to calculate the best possible performances for a measurement of the parameters that describe a general SU(d) unitary and we find an optimal scalable family of quantum states which achieves this performance for SU(3) unitaries. In our second project, we experimentally create and characterize the tetrahedron state in the polarization of four photons. This entangled state is the optimal 4-photon state for measuring the three parameters describing a SU(2) polarization rotation. Interestingly, it possesses the same rotational symmetries as a tetrahedron, giving the state its name. We perform state tomography on our experimental state and measure a quantum state fidelity of F = (0.46 ± 0.02) with the ideal state. Our state nevertheless shares many qualitative features with the latter. Finally, in our last project, we propose a technique named SPLICE which allows us to beat “Rayleigh’s curse”—the inability to measure the separation between two incoherent point sources of light when said separation is below the imaging apparatus’ Rayleigh’s criterion. We test this technique in a proof of principle experiment in the laboratory and demonstrate its advantage over traditional methods.

Keywords: Entanglement
Parameter estimation
Quantum metrology
Quantum optics
Super-resolution Imaging
Advisor: Steinberg, Aephraim M.
An Assessment of How Targeted Health Policies were used to Influence Access to Radiation Therapy Services in Ontario, Canada between 1997 and 2017

This research provides a broad understanding of the impact of policy on access to radiation therapy (RT) in Ontario. A suite of policies implemented over 20 years address RT wait times and utilization aspects of access to care. The overarching research question was: “What was the impact of provincial and cancer centre level health policies on access to RT in Ontario between 1997 and 2017?” A case study design with multiple embedded units was used. The case was Cancer Care Ontario (CCO), and the embedded units included four representative regional cancer centres. Methods included a document review, longitudinal quantitative data collection, and key informant interviews. The theoretical underpinning was an extension of Kingdon’s Multiple Streams Framework, which collectively examines the problem, solutions and politics surrounding an issue. The access to RT problem evolved from a wait times issue to a utilization issue. Thirty-seven policies were identified and categorized as: (1) improving existing RT capacity, (2) system planning, (3) performance management, (4) human resources, and (5) building new RT capacities. During the first decade, policies were reactive in design, resulting in short-term wait time improvements. Policies in the second decade were more proactive and strategic, providing longer-term solutions. Building new cancer centres yielded the largest sustainable improvement to RT access. Restructuring CCO’s role to an oversight body, and changes to funding mechanisms were necessary system planning policies that also supported access improvement. Politics was highly influential across all levels, with negative media press, public pressure, local politics, and a healthy economy as factors. The Ontario cancer systems’ Regional Vice President role operated as an effective policy entrepreneur. They were able to adapt provincial policy to implement innovative access solutions locally. This longitudinal study provides useful insights into the evolution of Ontario’s cancer system as a more close-knit community of stakeholders, allowing the approach to policy implementation to mature from more reactive to proactive. Access to care continues to be a government priority, particularly as the world struggles through the COVID-19 pandemic. Learnings from this study can inform policy implementation decision-making surrounding access to RT and more broadly cancer and health services.

Keywords: access to care
cancer services
Kingdon multiple streams framework
Ontario
policy
radiation therapy
Advisor: Dobrow, Mark