# Parametrisation of hybrid Gene Regulatory Networks using Artificial Evolution and Reinforcement Learning

Romain Michelucci, Denis Pallez, Jean-Paul Comet

► To cite this version:

Romain Michelucci, Denis Pallez, Jean-Paul Comet. Parametrisation of hybrid Gene Regulatory Networks using Artificial Evolution and Reinforcement Learning. 26ème congrès annuel de la société française de recherche opérationnelle et d'aide à la décision (ROADEF), ENPC, Institut Polytechnique de Paris, Feb 2025, Marne-la-Vallée, France. hal-04980785

HAL Id: hal-04980785

https://inria.hal.science/hal-04980785v1

Submitted on 6 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Parametrisation of hybrid Gene Regulatory Networks using
Artificial Evolution and Reinforcement Learning

Romain Michelucci, Denis Pallez, Jean-Paul Comet

Université Côte d'Azur, CNRS, I3S
{firstname.lastname}@univ-cotedazur.fr

**Keywords** : *Artificial evolution, MCTS, continuous GRAVE, multimodality, GRN.*

## 1 Introduction

Genetic regulatory network (GRN) modelling aims at studying and understanding the molecular mechanisms that enable the organism to perform essential functions ranging from metabolism to environmental disturbance adaptation. Studying the dynamics of these systems opens new perspectives with crucial applications in fundamental biology, pharmacology, medicine, or chronotherapy for instance, which tries to choose the best time of day to administer the medication in order to limit the side effects while preserving the therapeutic effects. The bottleneck in the modelling approach is the search for model parametrisations that are consistent with the available biological knowledge, often represented as irregularly spaced time series of observable events.

In this work, we have chosen *hybrid* frameworks (hGRN), which add to the discrete ones the time spent in each of the discrete states leading to the search of piecewise linear trajectories in a multidimensional space. These trajectories, each governed by a parametrisation, must respect the observed biological data, which have been previously and manually interpreted as a set of constraints based on Hoare logic [2]. A previous work logically applied continuous Constraint Satisfaction Problem (CSP) solvers to identify all valid trajectories [1], but faced difficulties in extracting solutions, especially when the number of genes exceeded 3 genes. Here, we first consider the determination of valid parameterisations as an unconstrained optimisation problem and solve it by comparing different metaheuristics. In a second step, we treat this problem as a sequential decision problem using reinforcement learning with Monte Carlo Tree Search (MCTS) algorithms. The two approaches are summed up in the next sections.

## 2 Unconstrained optimisation problem

We first transformed the CSP into an unconstrained minimisation problem by formulating a single-objective penalty fitness function [3] that computes how well a candidate solution satisfies the initial constraints. We then compared several metaheuristics such as DE, PSO, CMA-ES, GA and show that they are able to identify a solution equivalent to those found by continuous CSP. CMA-ES gives the best overall solutions, but GA seems to be the best metaheuristic due to its high probability of obtaining good results. Since a decrease in performance was observed with an increase in the number of genes, two coevolutionary optimisation frameworks were studied, Cooperative Coevolution and Group Evolution, with two decomposition strategies, by discrete state and by component. The results of these co-evolutionary optimisation solutions, which are still not as good as those of the GA, show the unsuitability of this type of framework for the problems studied, due to the non-separability of the problem and of the decomposition strategies of the co-evolution framework.

Compared to CSP, the main drawback of panmictic metaheuristics is that they tend to converge to a single basin of attraction in the search space, whereas biologists are interested in
finding multiple satisfactory solutions. In [4], we investigate the problem of multimodality and assesses the effectiveness of cellular GA (cGA) in dealing with the increasing dimensionality and complexity of hGRN models. A comparison with RS-CMSA-ESII, the winner of the CEC'2020 competition for multimodal optimisation (MMO), is conducted. Results show evidence that cGAs better maintain a diverse set of solutions while giving better quality solutions, making them better suited for this MMO task.

# 3 Sequential decision problem

In a second step, we model hGRN parameterisation problem as a sequential decision problem, represented by a Markov decision process with continuous states and actions, in order to solve it using MCTS. The MCTS selection function, GRAVE, combines the two well-known scores UCT and AMAF by considering the values of an ancestor node in the search tree to gather more accurate estimates near the leaves. In [6] we proposed to extend GRAVE to the continuous domain (cGRAVE) by adapting the AMAF statistics using Gaussian convolution smoothing, as cRAVE did for RAVE. Two improvements are also added, corresponding to a decomposition of a multidimensional action into several unidimensional actions and a pruning policy based on constraints to reduce the action search space. So far, different strategies have been tested in a cumulative manner on a real biological model which contains 5 genes. The results show the need for combining three previous enhancement to find solutions with a 100% success rate within a budget of 180k iterations. We also investigate multimodality in sequential decision problem, called diverse planning, in [5] by proposing three new strategies to enforce the diversity of the final set of decisions during the construction of the search tree.

# 4 Conclusion and Perspectives

We show that both approaches to parameterisation of hGRN using EA and RL have a larger applicability domain than the continuous CSP solver approach. We also show the shortcomings of the benchmark functions of the CEC'2020 competition on MMO. We propose to diversify the set of problems for this type of competition by adding these problems, whose specificities are not represented in the current competition. We have not yet compared the two approaches, as in the case of EA a complete and even unsatisfiable trajectory is evaluated, whereas in RL it is built up incrementally, making the comparison not so fair. For both approaches, future works should validate all proposed strategies on a more diverse set of problems. And finally, we expect to obtain better results by combining EA and RL.

# References

[1] J. Behaegel, J.-P. Comet, and M. Pelleau. "Identification of Dynamic Parameters for Gene Networks". In: *ICTAI*. 2018. DOI: 10.1109/ICTAI.2018.00028.

[2] G. Bernot et al. "A genetically modified Hoare logic". In: *Theoretical Computer Science* (2019). DOI: 10.1016/j.tcs.2018.02.003.

[3] R. Michelucci, J.-P. Comet, and D. Pallez. "Evolutionary continuous optimization of hybrid Gene Regulatory Networks". In: *EA*. 2022. DOI: 10.1007/978-3-031-42616-2_12.

[4] R. Michelucci et al. "Cellular Genetic Algorithms for identifying variables in hybrid Gene Regulatory Networks". In: *EvoAPPS*. 2024. DOI: 10.1007/978-3-031-56852-7_9.

[5] R. Michelucci et al. "Comparing diverse planning strategies with continuous MCTS applied to hGRN". In: *ICTAI*. 2024. DOI: 10.1109/ICTAI62512.2024.00018.

[6] R. Michelucci et al. "Improving Continuous Monte Carlo Tree Search for Identifying Parameters in Hybrid GRN". In: *PPSN*. 2024. DOI: 10.1007/978-3-031-70085-9_20.