title: Policy Search Based Relational Reinforcement Learning using the Cross-Entropy Method

authors: Sarjant, Samuel

subjects: relational reinforcement learning

abstract: Relational Reinforcement Learning (RRL) is a subfield of machine learning in which a learning agent seeks to maximise a numerical reward within an environment, represented as collections of objects and relations, by performing actions that interact with the environment. The relational representation allows more dynamic environment states than an attribute-based representation of reinforcement learning, but this flexibility also creates new problems such as a potentially infinite number of states.

This thesis describes an RRL algorithm named Cerrla that creates policies directly from a set of learned relational “condition-action” rules using the Cross-Entropy Method (CEM) to control policy creation. The CEM assigns each rule a sampling probability and gradually modifies these probabilities such that the randomly sampled policies consist of ‘better’ rules, resulting in larger rewards received. Rule creation is guided by an inferred partial model of the environment that defines: the minimal conditions needed to take an action, the possible specialisation conditions per rule, and a set of simplification rules to remove redundant and illegal rule conditions, resulting in compact, efficient, and comprehensible policies.

Cerrla is evaluated on four separate environments, where each environment has several different goals. Results show that compared to existing RRL algorithms, Cerrla is able to learn equal or better behaviour in less time on the standard RRL environment. On other larger, more complex environments, it can learn behaviour that is competitive to specialised approaches. The simplified rules and CEM’s bias towards compact policies result in comprehensive and effective relational policies created in a relatively short amount of time.

text: Policy Search Based Relational Reinforcement Learning using the Cross-Entropy Method Relational Reinforcement Learning (RRL) is a subfield of machine learning in which a learning agent seeks to maximise a numerical reward within an environment, represented as collections of objects and relations, by performing actions that interact with the environment. The relational representation allows more dynamic environment states than an attribute-based representation of reinforcement learning, but this flexibility also creates new problems such as a potentially infinite number of states.

This thesis describes an RRL algorithm named Cerrla that creates policies directly from a set of learned relational “condition-action” rules using the Cross-Entropy Method (CEM) to control policy creation. The CEM assigns each rule a sampling probability and gradually modifies these probabilities such that the randomly sampled policies consist of ‘better’ rules, resulting in larger rewards received. Rule creation is guided by an inferred partial model of the environment that defines: the minimal conditions needed to take an action, the possible specialisation conditions per rule, and a set of simplification rules to remove redundant and illegal rule conditions, resulting in compact, efficient, and comprehensible policies.

Cerrla is evaluated on four separate environments, where each environment has several different goals. Results show that compared to existing RRL algorithms, Cerrla is able to learn equal or better behaviour in less time on the standard RRL environment. On other larger, more complex environments, it can learn behaviour that is competitive to specialised approaches. The simplified rules and CEM’s bias towards compact policies result in comprehensive and effective relational policies created in a relatively short amount of time.

year: 2013

title: Invasive European rats in Britain and New Zealand: same species, different outcomes

authors: King, Carolyn M.

subjects: Rattus rattus

abstract: Two species of European commensal murids, the Norway rat norvegicus and the ship rat Rattus rattus, have colonized two island archipelagos of comparable size and temperate climate but in opposite hemispheres and with opposite outcomes. Ship rats were common commensals in Britain until replaced by Norway rats; Norway rats were hugely abundant in native forests throughout New Zealand until widely replaced by ship rats. Interference explains the first case, as ship rats are smaller than Norway rats and are always vulnerable to aggressive competition from them, but some other explanation is needed for the second case. We used the marginal value theorem to investigate exploitation competition between these two species in arboreal habitats. We observed the climbing behaviour and ‘giving-up time’ of captive rats of both species searching for food at different heights above the ground. Our data confirmed that the smaller size and greater agility of R. rattus give it a competitive advantage in foraging for scattered small food items above ground. We propose that (1) the outcomes of the interactions between the two rat species in any given place depend on the distribution of food resources in structurally complex habitat, moderated by winter temperatures; (2) the different outcomes of invasions by the two species can be explained in Britain by interference competition, and in New Zealand by exploitation competition and by the absence of specialist arboreal rodents (squirrels).

text: Invasive European rats in Britain and New Zealand: same species, different outcomes Two species of European commensal murids, the Norway rat norvegicus and the ship rat Rattus rattus, have colonized two island archipelagos of comparable size and temperate climate but in opposite hemispheres and with opposite outcomes. Ship rats were common commensals in Britain until replaced by Norway rats; Norway rats were hugely abundant in native forests throughout New Zealand until widely replaced by ship rats. Interference explains the first case, as ship rats are smaller than Norway rats and are always vulnerable to aggressive competition from them, but some other explanation is needed for the second case. We used the marginal value theorem to investigate exploitation competition between these two species in arboreal habitats. We observed the climbing behaviour and ‘giving-up time’ of captive rats of both species searching for food at different heights above the ground. Our data confirmed that the smaller size and greater agility of R. rattus give it a competitive advantage in foraging for scattered small food items above ground. We propose that (1) the outcomes of the interactions between the two rat species in any given place depend on the distribution of food resources in structurally complex habitat, moderated by winter temperatures; (2) the different outcomes of invasions by the two species can be explained in Britain by interference competition, and in New Zealand by exploitation competition and by the absence of specialist arboreal rodents (squirrels).

year: 2011

title: Utuhina Stream 2006-2020: In-stream alum dosing effects on fish and aquatic invertebrates

authors: Ling, Nicholas

abstract: This report presents the results of an on-going assessment of the fish and aquatic macroinvertebrate communities of the Utuhina Stream, Lake Rotorua, from 2006 to 2020, and an assessment of the bioavailability of aluminium in fish and kōura to satisfy Bay of Plenty Regional Council’s annual resource consent conditions 9.6, 9.8 and 9.7, respectively, for consent 65321 for the discharge of alum to the Utuhina Stream. Macroinvertebrates, fish and kōura (freshwater crayfish) were sampled from one control and two treatment reaches of the Utuhina Stream annually. Common bully (Gobiomorphus cotidianus) is the dominant species in the fish community of the Utuhina Stream. Kōura (Paranephrops planifrons) and juvenile trout were always present at all sites but variable in abundance. Differences in species abundance from year to year are most likely due to flood-related disturbances to stream bank morphology and in-stream vegetative cover or physical displacement of fish. No obvious effects of alum dosing on stream fish or macroinvertebrate communities were observed between the upstream control site and sites downstream of the alum discharge. Several other fish species were occasionally captured during sampling and the regular occurrence of juvenile koaro from 2016 to 2019 is possible evidence of this taonga species becoming established in the Utuhina. Analysis of stream macroinvertebrates also showed no consistent differences between the upstream control site and the sites downstream of the alum dosing. Overall, all sites were characterised as fair to good quality for a soft-bottomed stream. Some evidence of aluminium bioaccumulation was seen in some tissues of common bully (gills and liver) in some years, resulting from continuous alum dosing of the Utuhina Stream, but there was no evidence of bioaccumulation of aluminium in the tissues of kōura. Alum exposure in these species does not appear to affect their health or abundance in the stream.   iii Overall, continuous alum discharge does not appear to negatively impact the ecology of the lower Utuhina and improvements in the ecological condition of the Utuhina Stream will be achieved by ongoing riparian restoration and mitigation of the impacts of flood flows.

text: Utuhina Stream 2006-2020: In-stream alum dosing effects on fish and aquatic invertebrates This report presents the results of an on-going assessment of the fish and aquatic macroinvertebrate communities of the Utuhina Stream, Lake Rotorua, from 2006 to 2020, and an assessment of the bioavailability of aluminium in fish and kōura to satisfy Bay of Plenty Regional Council’s annual resource consent conditions 9.6, 9.8 and 9.7, respectively, for consent 65321 for the discharge of alum to the Utuhina Stream. Macroinvertebrates, fish and kōura (freshwater crayfish) were sampled from one control and two treatment reaches of the Utuhina Stream annually. Common bully (Gobiomorphus cotidianus) is the dominant species in the fish community of the Utuhina Stream. Kōura (Paranephrops planifrons) and juvenile trout were always present at all sites but variable in abundance. Differences in species abundance from year to year are most likely due to flood-related disturbances to stream bank morphology and in-stream vegetative cover or physical displacement of fish. No obvious effects of alum dosing on stream fish or macroinvertebrate communities were observed between the upstream control site and sites downstream of the alum discharge. Several other fish species were occasionally captured during sampling and the regular occurrence of juvenile koaro from 2016 to 2019 is possible evidence of this taonga species becoming established in the Utuhina. Analysis of stream macroinvertebrates also showed no consistent differences between the upstream control site and the sites downstream of the alum dosing. Overall, all sites were characterised as fair to good quality for a soft-bottomed stream. Some evidence of aluminium bioaccumulation was seen in some tissues of common bully (gills and liver) in some years, resulting from continuous alum dosing of the Utuhina Stream, but there was no evidence of bioaccumulation of aluminium in the tissues of kōura. Alum exposure in these species does not appear to affect their health or abundance in the stream.   iii Overall, continuous alum discharge does not appear to negatively impact the ecology of the lower Utuhina and improvements in the ecological condition of the Utuhina Stream will be achieved by ongoing riparian restoration and mitigation of the impacts of flood flows.

year: 2021-06