Parameter Inference with Bifurcation Diagrams

1 Introduction . Inverse problems [ 1 ] arise in biology and engineering in settings when the model is not fully known and the desire is to match model behaviour to a given set of observations . This helps systematically guide both model and experimental design . While we would like to understand the quantitative details of a system , often only qualitative changes in response to varying experimental conditions can be robustly measured across independent studies [ 2 , 3 ] . For example , several studies are likely to agree that the human immune system activates above a threshold concentration of a pathogen and deactivates at a lower threshold concentration , but may disagree on the exact quantities of the thresholds or the magnitudes of the immune response . Bifurcation theory provides us a framework for studying these transitions in a manner that is independent of quantitative details [ 4 ] . The emerging picture suggests that identification of the qualitative behaviour – the bifurcation diagram – should precede any attempt at inferring other properties of a system [ 5 ] . Inferring the parameters of a model directly from a bifurcation diagram is difficult because it is not obvious how multiple parameters in concert control the existence and position of a bifurcation . It could even be impossible for the model to bifurcate in the manner desired . For models with a sufficiently small number of parameters , finding specific bifurcation diagrams is typically done by 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . hand [ 6 ] . Several approaches exist to place bifurcations to desired locations once a manifold is present [ 7–9 ] yet typically resort to sampling techniques to search for them in the first place [ 10 , 11 ] . It is always possible to design bespoke goodness of fit measures to find specific model behaviours , for example using the period and phase of limit cycle oscillations [ 12 ] . However , this approach does not generalise across a wider set of qualitative behaviours . Progress has been made in cases where model structure and stability conditions are used to refine the search space [ 13 , 14 ] yet the resulting objectives are still not explicit in the bifurcation targets and also not differentiable . In the emerging field of scientific machine learning [ 15–17 ] , parameters of structured mechanistic models are favoured over flexible models in larger parameter spaces . A scalable method for navigating the space of bifurcation diagrams would enable design of differential equations with high-level qualitative constraints . Furthermore one could begin organising models according to qualitatively distinct behaviours . Back-propagation through differential equation solvers has been a breakthrough over the past couple of years [ 18 , 19 ] that enabled scalable parameter inference for differential equations from trajectory data . Although one could use trajectory data to create the aforementioned qualitative constraints [ 20 , 21 ] this would entail over-constraining information originating from the kinetics and dynamical transients of the model . Furthermore , such data usually does not contain sufficient information about dynamical transients in order to identify kinetic parameters . Techniques for back-propagating through implicit equation solvers have also been developed [ 22 , 23 ] although to the best of the authors ’ knowledge have not been applied to bifurcation diagrams at the time of writing this paper . The problem of inferring differential equation parameters against a user-specified bifurcation diagram decomposes into two parts : searching for bifurcating regimes and matching the locations of bifurcation points to desired values . Matching bifurcation locations is a supervised problem where the data are expressed as bifurcations points [ 8 , 11 ] . Searching for bifurcations is an unsupervised problem because when bifurcations are not present , there is no distance defined between data and prediction [ 10 ] . Therefore only properties of the model can be used to start the search . We propose an approach for performing both tasks in an end-to-end fashion . The bifurcation diagram encodes high-level qualitative information defined by state space structures , rather than kinetics . We apply the strategy of implicit layers [ 22 , 23 ] to calculate gradients . To compute the diagram we use a predictor-corrector method called deflated continuation [ 24 , 25 ] developed for partial differential equations . We find that the cost function landscape contains basins that not only allow us to synthesise models with a desired bifurcation diagram but also allow us to organise models in terms of topological and geometric equivalence . We discuss the relevance of this in model selection . In summary , our paper has the following main contributions : • An end-to-end differentiable method for locating bifurcations in parameter space and then matching their dependency on a control condition to user-specified locations • Implementation of the method as a Julia package BifurcationInference.jl • Leveraging the cost landscape for a novel way of organising differential equation models in terms of geometric and topological equivalence 1.1 Preliminaries . Suppose we collected observations along a scalar control condition p ∈ R and conclude that there are specific values of p for which there are qualitative changes in system behaviour . Let D be the set of those values and let us hypothesise that these transitions occur due to bifurcations in the dynamics that drive the underlying mechanism . Let us model the mechanism with a parametrised set of differential equations for states u ∈ RN with a vector function Fθ in a parameter space θ ∈ RM . For the purposes of introducing this work , we will consider the simplest class of bifurcations known as co-dimension one bifurcations not including limit cycles . Therefore D should contain conditions for which we hypothesise changes in multi-stable behaviour . Let the equations be ∂u ∂t = Fθ ( u , p ) where Fθ : RN+1 → RN ( 1 ) In the context of the differential equations , and not considering limit cycles for now , we show that a static non-degenerate bifurcation can be defined by a set of conditions on the determinant of the Jacobian ∣∣∂Fθ ∂u ∣∣ . The determinant of the Jacobian quantifies the rate at which trajectories in a local patch of state-space u ∈ RN converge or diverge . Let s ∈ R parametrise the curves that trace out the bifurcation diagram . Any location on the curve u ( s ) and p ( s ) must satisfy the steady-state of equations ( 1 ) . Directional derivatives dds along the diagram require the calculation of a vector that is tangent to the diagram ( see Supplementary A ) . The determinant approaching zero along the diagram means that the dynamics of the system are slowing down , which is an important indicator for the onset of a transition between qualitative behaviours . Furthermore , the slowing down must necessarily be followed by a breakdown of stability ; for this to be true it is sufficient but not necessary to require that the determinant cross zero with a finite slope , meaning that its directional derivative along the diagram dds ∣∣∂Fθ ∂u ∣∣ is not zero . This is the non-degeneracy condition . The set of predicted values for the control condition P ( θ ) ⊂ R at which bifurcations occur are defined as P ( θ ) : = { p | ∃ u : Fθ ( u , p ) = 0 , ∣∣∣∣∂Fθ∂u ∣∣∣∣ = 0 , dds ∣∣∣∣∂Fθ∂u ∣∣∣∣ 6= 0 } ( 2 ) A proof of how the conditions ( 2 ) are necessary and sufficient for static non-degenerate bifurcations is detailed in Supplementary B . The most common bifurcations between steady states , not including limit cycles , are saddle-nodes and pitchforks [ 26 ] . Saddle-node bifurcations , which often appear in pairs ( Figure 1A ) are defined by stable and unstable fixed points meeting and disappearing . Pitchfork bifurcations occur where a single steady state splits into two stable and one unstable steady state ( Figure 1B shows an imperfect pitchfork ; a perfect pitchfork arises when θ1 = 0 ) . To illustrate these bifurcations , we define minimal models ( Figure 1 ) that span the space of saddle-node and pitchforks , where indeed zero crossings in the determinant with a finite slope define the set of prediction P ( θ ) . The location of these crossings in general may not match the targets D. For a given set of parameters θ one could compute the set of predicted bifurcations P ( θ ) using parameter continuation methods [ 25 , 24 ] . Our goal is to find optimal parameters θ∗ that match predictions P ( θ∗ ) to specified targets D. We must design a suitable cost function L so that θ∗ : = argminθL ( θ|D ) ( 3 ) The optimal θ∗ is not expected to always be unique , but is in general a manifold representing the space of qualitatively equivalent models . Ideally , the cost function L should reward θ for which the number of predicted bifurcations is equal to the number of targets , |P ( θ ) | = |D| . This is especially important in the case where there are no predictions |P ( θ ) | = 0 . 2 Proposed Method . 2.1 Cost Function . To identify parameter sets that give rise to bifurcation diagrams with specified bifurcation points , we propose a cost function that comprises two terms . The role of the error term is simply to reward predicted bifurcations to coincide with the specified target locations . This of course relies on such bifurcations existing . The role of the eigenvalue term is to encourage an optimiser to move towards parameter regimes that do exhibit bifurcations . 2.1.1 Error term : matching bifurcations to target locations . In order for predicted bifurcations p ( θ ) ∈ P ( θ ) to match targets p′ ∈ D we need to evaluate an error term |p ( θ ) −p′| . A naive approach might take an average over the norms for all prediction-target pairs . However this gives rise to unwanted cross-terms and the possibility of multiple predictions matching the same target without any penalty for unmatched targets . Therefore , we choose a geometric mean over the predictions and an arithmetic mean over targets : E ( θ , D ) = 1 |D| ∑ p′∈D ∏ p ( θ ) ∈P ( θ ) |p ( θ ) − p′| 1 |P| ( 4 ) The error term is only zero when each target is matched by at least one prediction and allows for cases where the number of predictions is greater than or equal to the number of targets |P| ≥ |D| . An alternative approach , which undesirably introduces more hyper-parameters , would be to let each prediction P ( θ ) represent the centroid of a mixture distribution and use expectation-maximisation to match the centroids to targets D . 2.1.2 Eigenvalue term : encouraging bifurcations . We can see from Figure 1 and definition ( 2 ) that predictions p ( θ ) can be identified by looking for points along the curve where the determinant crosses zero ∣∣∂Fθ ∂u ∣∣ = 0 with a finite slope d ds ∣∣∂Fθ ∂u ∣∣ 6= 0 . Using these quantities we can define a positive semi-definite measure ϕθ ( s ) of zero crossings in the determinant along a curve parametrised by s which we define as ϕθ ( s ) : = ( 1 + ∣∣∣∣∣ ∣∣∂Fθ ∂u ∣∣ d ds ∣∣∂Fθ ∂u ∣∣ ∣∣∣∣∣ ) −1 ( 5 ) The bifurcation measure ϕθ ( s ) is maximal at bifurcations and has finite gradients in nonbifurcating regimes ( Figure 2 ) . More specifically , the measure ϕθ ( s ) is one at bifurcation points and goes to zero an odd number of times between bifurcations . This is because ∣∣∂Fθ ∂u ∣∣ must eventually turn around in order to return back to zero , resulting in the directional derivative dds ∣∣∂Fθ ∂u ∣∣ going to zero . Hence the measure ϕθ ( s ) goes to zero for each turning point ( see Figure 2 ) . On the other hand , as the determinant ∣∣∂Fθ ∂u ∣∣ diverges , we approach regimes far away from any bifurcations and hence ϕθ ( s ) → 0 . Since we would still like to have non-zero gradients with respect to θ in these regimes we designed the measure to go to zero sufficiently slowly . While the calculation of the determinant is straightforward , its directional derivative requires a tangent vector to the bifurcation curve . Fortunately the tangent vector Tθ ( s ) at the solution u ( s ) , p ( s ) anywhere along the curve s can be calculated as the nullspace of the rectangular N × ( N + 1 ) Jacobian ∂Fθ ∂ ( u , p ) ∣∣∣∣ Fθ ( u ( s ) , p ( s ) ) =0 · Tθ ( s ) = 0 ( 6 ) This equation guarantees that the tangent vector Tθ ( s ) is orthogonal to all hyper-planes defined by the components of Fθ . In this setting the dimension of the nullspace is always known , and therefore can reliably be calculated using QR factorisation methods [ 27 ] . Equipped with a measure that quantifies the appearance of bifurcations along a bifurcation arc we can define the total measure for a bifurcation diagram as Ψ ( θ ) : = ∫ Fθ ( u , p ) =0 ϕθ ( s ) ds∫ Fθ ( u , p ) =0 ds . ( 7 ) Here we denote ∫ Fθ ( u , p ) =0 ds as the sum of the line integrals in ( u , p ) ∈ RN+1 defined by the level set Fθ ( u , p ) = 0 with s being an arbitrary parametrisation of the curves . The total measure Ψ ( θ ) is normalised such that Ψ ( θ ) → 1 in the regimes where the controlled condition region p is densely packed with bifurcations . The total measure Ψ ( θ ) is added to the error term as if it were a likelihood . This defines the cost function as L ( θ|D ) : = ( |P| − |D| ) log Ψ ( θ ) + E ( θ , D ) , ( 8 ) The pre-factor |D| − |P| in the eigenvalue term ensures that the gradients are always pushing optimisers towards a state where |D| = |P| . This can be seen as a step-wise annealing of the eigenvalue term until the desired state is reached .

Summary: In this work, authors suggest a gradient-based semi-supervised approach to infer the parameters of ODEs for producing a bifurcation diagram. A bifurcation measure is defined that uses the determinant of the state-space Jacobian as an indicator for bifurcating parameter regimes in the unsupervised term of the cost function. They demonstrate parameter inference with minimal models which explore the space of saddle-node and pitchfork diagrams and the genetic toggle switch from synthetic biology.