# Setting the basis of best practices and standards for curation and annotation of logical models in biology-highlights of the [BC]2 2019 CoLoMoTo/SysMod Workshop

Anna Niarakis, Martin Kuiper, Marek Ostaszewski, Rahuman Malik Sheriff, Cristina Casals-Casas, Denis Thieffry, Tom Freeman, Paul Thomas, Vasundra Touré, Vincent Noël, et al.

► To cite this version:

Anna Niarakis, Martin Kuiper, Marek Ostaszewski, Rahuman Malik Sheriff, Cristina Casals-Casas, et al.. Setting the basis of best practices and standards for curation and annotation of logical models in biology-highlights of the [BC]2 2019 CoLoMoTo/SysMod Workshop. Briefings in Bioinformatics, 2021, 22 (2), pp.1848-1859. 10.1093/bib/bbaa046. hal-02549796

HAL Id: hal-02549796

https://inria.hal.science/hal-02549796

Submitted on 31 Jul 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Setting the basis of best practices and standards for curation and
annotation of logical models in biology – Highlights of the [BC]2 2019
CoLoMoTo/SysMod Workshop

Anna Niarakis, Martin Kuiper, Marek Ostaszewski, Rahuman S Malik Sheriff, Cristina

Casals-Casas, Denis Thieffry, Tom C. Freeman, Paul Thomas, Vasundra Touré, Vincent

Noel, Gautier Stoll, Julio Saez-Rodriguez, Aurelien Naldi, Eugenia Oshurko, Ioannis

Xenarios, Sylvain Soliman, Claudine Chaouiya, Tomáš Helikar and Laurence Calzone

Authors Affiliations

Anna Niarakis GenHotel, Univ. Évry, Université Paris-Saclay, Genopole, 91025 Évry, France

Martin Kuiper, Department of Biology, Norwegian University of Science and Technology (NTNU),
Trondheim, Norway

Marek Ostaszewski- Luxembourg Centre for Systems Biomedicine, University of Luxembourg, 6
Avenue du Swing, L-4367 Belvaux, Luxembourg

Rahuman S. Malik Sheriff, European Bioinformatics Institute (EMBL-EBI), Cambridge, United
Kingdom

Cristina Casals-Casas, Swiss-Prot group, SIB Swiss Institute of Bioinformatics, 1 Michel Servet, 1211

Geneva 4, Switzerland

Paul Thomas - GO, (GO-CAM/Noctua), USC, USA

Denis Thieffry, Computational Systems Biology Team, Institut de Biologie de l'École Normale

Supérieure (IBENS), CNRS, INSERM, École Normale Supérieure, Université PSL, Paris, France

Tom C. Freeman, The Roslin Institute and Royal (Dick) School of Veterinary Studies, University of
Edinburgh, Easter Bush, Edinburgh, EH25 9RG, United Kingdom

Vasundra Touré, Department of Biology, Norwegian University of Science and Technology (NTNU),
Trondheim, Norway

Vincent Noël, Institut Curie / U900 INSERM / Mines ParisTech, France

Gautier Stoll, Team of Metabolism, Cancer & Immunity, Centre de Recherche des Cordeliers,
INSERM UMRS1138, Sorbonne Université, Université Paris Descartes, Université Paris Diderot,
Paris, France; Metabolomics and Cell Biology Platforms, Gustave Roussy Cancer Campus, Villejuif,
France
Julio Saez-Rodriguez: Institute for Computational Biomedicine, Heidelberg University, Faculty of Medicine, & Heidelberg University Hospital, Bioquant - Room 464 Im Neuenheimer Feld 267, 69120 Heidelberg, Germany

JRC for Computational Biomedicine, RWTH Aachen, Faculty of Medicine, Aachen, Germany
Aurélien Naldi, Computational Systems Biology Team, Institut de Biologie de l'École Normale Supérieure (IBENS), CNRS, INSERM, École Normale Supérieure, Université PSL, Paris, France

Eugenia Oshurko, Univ Lyon, EnsL, UCBL, CNRS, LIP, F-69342 LYON Cedex 07, France
Ioannis Xenarios, CHUV Agora Center, UNIL, Ludwig Institute of Cancer Research, Rue du Bugnon 25A, Lausanne, Switzerland

Sylvain Soliman, Lifeware group, Inria Saclay-Île de France, 91120 Palaiseau, France

Claudine Chaouiya, Aix Marseille Univ, CNRS, Centrale Marseille, I2M, Marseille, France

Tomas Helikar, Department of Biochemistry, University of Nebraska-Lincoln Lincoln, NE, USA

Laurence Calzone, Institut Curie, PSL Research University, Mines ParisTech, Inserm, U900, F-75005, Paris, France

## Abstract

The fast accumulation of biological data calls for their integration, analysis and exploitation through more systematic approaches. The generation of novel, relevant hypotheses from this enormous quantity of data remains challenging. Logical models have long been used to answer a variety of questions regarding the dynamical behaviours of regulatory networks. As the number of published logical models increases, there is a pressing need for systematic model annotation, referencing and curation in community-supported and standardised formats. This article summarizes the key topics and future directions of a meeting entitled "Annotation and curation of computational models in biology", organized as part of 2019 [BC]2 conference. The purpose of the meeting was to develop and drive forward a plan towards the standardised annotation of logical models, review and connect various ongoing projects of experts from different communities involved in modelling and annotation of molecular biological entities, interactions, pathways and models. This article defines a
roadmap towards annotation and curation of logical models, including milestones for best practices and minimum standard requirements.

**Keywords:** biocuration, logical modelling, reproducibility, model reusability, annotation standards

## Introduction

Reproducibility of research findings constitutes a key concern of the scientific community as multiple reports show that published results in various scientific domains cannot be replicated [1]. In the field of computational systems biology, where scientists combine prior knowledge based on experimental evidence and computational approaches, the reproducibility of results can be fostered through the use of consensual practices and standards, extensive annotation, code sharing, as well as by depositing of the resulting models in dedicated repositories.

Logical (or logic) models (Boolean, multivalued, or other variants) have been widely used for studying and analysing in-depth regulatory mechanisms and biological processes for which kinetic data are scarce. Some repositories for this type of models exist already, including GINsim repository [2] and Cell Collective, a platform for building, analysing and visualising models [3,4].

In the GINsim repository, one can find models built with the software GINsim and used for simulations in peer-reviewed articles. Models are stored in their zginml format and a summary along with a link to the supporting scientific article are provided. In Cell Collective, models have been manually curated by re-construction, re-simulation and analysis to ensure that their dynamics correspond to published results. Efforts are further made to include logical models in BioModels, a repository of mathematical models of biological and
biomedical systems [5]. Annotation practices, accuracy and reproducibility checks made by
the BioModels team will facilitate consistent quality control of these models.

To facilitate exchanges of logical models and communication between tools, previous work
by the CoLoMoTo consortium and Systems Biology Markup Language (SBML) teams
focused on standardisation of model formats by developing a specific package of the Systems
Biology Markup Language level 3 (SBML L3) [6], SBML-qual [7,8].

However, specific minimum requirements for the annotation and level of curation of logical
models remain to be defined. Even when results are reproducible, models often fail to be
reusable because of the lack of explicit referencing to the sources used for their construction
(organism, experimental setting and type of data, published manuscript sources, identifiers to
relevant database entries, etc.).

To address the pressing need to propose and develop best practices and standards in
annotation and curation of logical models in biology, Anna Niarakis, Laurence Calzone and
Tomáš Helikar (representatives of the CoLoMoTo [9] and SysMod [10] communities)
organized a workshop in the context of the [BC]² conference recently held in Basel [11], with
the aim to bring together logical modellers and curators. The workshop, entitled "Annotation
and curation of computational models in biology" [12] is the most recent of a series of
workshops organised by the logical modelling community over the past years, in the context
of prominent international conferences such as ECCB 2014 (Strasbourg, France), ICSB 2015
(Singapore), ECMTB 2016 (Nottingham, UK), [BC]2 2017 (Basel, Switzerland), ECCB 2018
(Athens, Greece).

The meeting was divided into four sessions highlighting key challenges of the modelling
community (Figure 1), starting with curation platforms and model repositories. In particular,
the need for establishing annotation criteria, quality control checks and the use of a common
repository were extensively discussed. The following session focused on recent methodological advancements to analyse logical models to ensure interoperability and reusability, and lastly, the afternoon sessions were focused on integrative approaches and tools. In Table 1, we have summarized briefly the topics discussed in each session. The presentations were followed by an extensive discussion between all speakers and participants on three key topics:

**Reproducibility**, i.e., the ability to replicate scientific results using the same model.

**Reusability**, i.e., the ability to reuse an existing model using transparent biocuration processes, extensive annotations and references that increase the model's liability.

**Interoperability**, i.e., the ability to analyse the same model with multiple tools due to the use of standard formats.

<table><caption>**Table 1** Summary of different topics and presentations.</caption><thead><tr><th>Workshop sessions and Chairs</th><th>Presentations and speakers</th></tr></thead><tbody><tr><td>Model curation and annotation and available repositories<br><br>Chairs:<br><br>Anna Niarakis and Denis Thieffry</td><td><ul><li><strong>Martin Kuiper</strong>: Towards a curation platform for causal interaction statements.</li><li><strong>Marek Ostaszewski</strong>: BioKB and MINERVA: a workflow for curation and quick prototyping of annotated knowledge repositories</li><li><strong>Rahuman S Malik Sheriff</strong>: Curation and annotation of models in BioModels repository promotes reproducibility and reusability</li><li><strong>Cristina Casals</strong>: SysVasc Prior Knowledge Network: An example of biocuration for Boolean modelling</li></ul></td></tr></tbody></table>
**Community standards**
**development and**
**interoperability/reusability**

**Chairs:**

Marek Ostaszewski and
Laurence Calzone

**Tools (I)**

Chair Julio Saez
Rodriguez

**Tools (II)**

Chair Tomas Helikar

* **Denis Thieffry:** Computational verification of large logical models - application to the prediction of T cell response to checkpoint inhibitors

* **Tom Freeman:** A graphical and computational model of the renal mammalian circadian clock

* **Paul Thomas:** Gene Ontology Causal Activity Modeling

* **Anna Niarakis:** Automated inference of annotated Boolean models from molecular interaction maps using CaSQ

* **Tomas Helikar:** Cell Collective modelling platform

* **Gaultier Stoll and Vincent Noel:** MaBoSS ecosystem

* **Vasundra Touré:** The Minimum Information about a Molecular Interaction Causal Statement (MI2CAST): a guideline for the annotation of molecular causal interactions

* **Julio Saez Rodriguez** CellNOpt

* **Aurélien Naldi:** The CoLoMoTo Interactive Notebook: Accessible and Reproducible Computational Analyses for Qualitative Biological Networks

* **Eugenia Oshurko:** KAMISudio

## Model curation and annotation, and available repositories

The first session was dedicated to annotation and curation approaches, together with relevant repositories, including the presentation of curation approaches and tools for the development of Boolean models for colon cancer and molecular causal interaction statements, the introduction to the complementary platforms BioKB [13] and MINERVA [14], followed by that of the BioModels repository. An example of an atherosclerotic plaque formation model
demonstrated the necessity of proper annotation for optimal model-based predictions. The first session highlighted the necessity to annotate prior knowledge networks (PKNs) and logical models accurately for reusability, and enrich them with knowledge from heterogeneous resources to avoid potential ambiguities (e.g., UniProtKB [15], SIGNOR [16], HGNC [17], GO [18], REACTOME [19]).

Martin Kuiper (DrugLogics team, NTNU) presented work on a set of Colon Cancer logical models named CASCADE (CAncer Signaling CAusality DatabasE, [20]), and the development of a novel curation interface named Visual Syntax Method (VSM, [21]), which enables the curation of biological network information that includes causal molecular relationships. The VSM interface was tested extensively to annotate the full collection of experimentally analysed DNA binding transcription factors for human, mouse and rat [22], and is now being implemented in a *curation platform for causal interaction statements* [23].

Causal interaction statements are basic representations of regulatory interactions between two biological entities that can be efficiently extracted from the literature, provided that proper annotation tools and curation guidelines are provided.

Marek Ostaszewski (Luxembourg Centre for Systems Biomedicine) presented *BioKB and MINERVA: a workflow for curation and fast prototyping of annotated knowledge repositories* [13,14]. To construct graphical models of molecular mechanisms, one needs to i) extract entities, interactions and relevant annotations from the literature, ii) build a consistent graphical representation, and iii) review and parameterise the model. BioKB [13,24] is a platform initially designed for exploring text mining data, which currently allows combining human-provided and machine-identified elements and their interactions into “facts” – human-curated relationships, annotated with sentences, literature and recognized identifiers. As BioKB is not a diagram editor, the biocurator can focus only on the accuracy of the
extracted facts. This model visualization step, however, can be complemented with the
MINERVA Platform, which allows API-driven [25] conversion of a layout-less model into
an editable diagram (SBGN-ML, [26]) that can be further used by various systems biology
editors (e.g., CellDesigner [27], Newt [28], etc.). This way also the final step of the model
curation workflow can be realised - a curated diagram can be exported to the chosen systems
biology format, refined and parameterised. Moreover, such API-based conversion makes it
convenient to include in bigger bioinformatic workflows.

Following the effort towards transparency of the different steps leading to model construction
and the reusability of these models, Rahuman S. Malik-Sheriff (European Bioinformatics
Institute (EMBL-EBI)) discussed how *Curation and annotation of models in BioModels
repository promote reproducibility and reusability*. Established in 2005, BioModels provides
a platform to support sharing, easy accessibility and reproducibility of mathematical models
of biological processes [5,29]. Models submitted to BioModels are verified and curated by
expert in-house curators. In 2011, an effort was made to extend the standard to logical
formalism and SBML-*qual* was defined [7,8], allowing the inclusion of logical models in the
database. Following *Minimum Information Requested In the Annotation of Models*
(MIRIAM) guidelines, curated models are encoded in the standard SBML format and
semantically enriched with controlled vocabularies [30]. Model entities are linked to several
data resources (e.g., UniProt [15], Ensembl [31], the NCBI Taxonomy Database [32]), as
well as ontologies, such as Gene Ontology [18], ChEBI [33], Mathematical Modelling
Ontology [34], Systems Biology Ontology [35], and Brenda Tissue Ontology [36]. Such
annotations allow the unambiguous identification of model components and processes.
BioModels currently hosts over 900 curated models, becoming the world's largest repository
of curated models. BioModels team will soon start to systematically curate logical models.
To date, however, only seventeen logical models, three curated and fourteen non-curated are included in the BioModels' collection.

Cristina Casals-Casas from (Swiss-Prot) presented *SysVasc Prior Knowledge Network (PKN): An example of biocuration for dynamical modelling*. As a case study, Cristina Casals-Casas and collaborators have built a PKN to allow dynamical modelling of atherosclerotic plaque formation [37]. The expert curation strategy was centred on regulatory interactions between biological entities (gene products, chemical compounds and processes) interacting with each other in a complex manner, and exhibiting conditional dependencies between co-regulators. Biological entities were defined using strictly controlled vocabulary terms, retrieved from UniProtKB, HGNC, ChEBI, or GO, among others. The resulting PKN includes 729 components linked by 3,406 interactions of which 1,841 are complex regulations encoded with logical operators, while 1,565 are simply activatory or inhibitory interactions. For each component, they demonstrated how the description of complex signalling functions and their integration are essential to correctly predict activation state in health and disease states. Their work highlighted the essential role of expert curation to correctly identify and encode complex regulatory interactions from experimental literature. Failure to encode these relationships correctly can alter significantly the behaviour of the model and the derived predictions. Dynamical models should be fine-tuned by contextualization to the specific biological system under study, and for this, proper annotation and expert curation are essential.

**Community standards development and interoperability/reusability of
existing models**
The second session of the meeting was dedicated to interoperability and reusability of models and provided examples using three different model applications. Novel dynamical analysis methods and a framework for Gene Ontology annotations for supporting model building were also presented. All these approaches take advantage of existing databases to assist modellers and automatise error-prone and cumbersome tasks, currently still often performed manually, in order to optimise iterative modelling.

Denis Thieffry (Ecole Normale Supérieure, Paris) presented novel methods for the *Computational verification of large logical models*, with an *application to the prediction of T cell response to checkpoint inhibitors*. A first approach enables the formalisation and automatic verification of validation criteria for whole models or defined subparts, thereby greatly facilitating model development and correction. A second approach consists in computing the impact of specific environmental or genetic perturbations on model dynamics by propagating their impact on model logical rules. These methods were applied to the analysis of the impact of T lymphocyte checkpoint inhibitors and their use was integrated and illustrated in the CoLoMoTo Interactive Notebook [38] (presented by Aurelien Naldi in the afternoon session) to foster transparent and reproducible analyses.

Tom Freeman (Roslin Institute) presented a *graphical and computational model of the renal mammalian circadian clock*. A comprehensive graphical model of the circadian pathway was constructed using the modified Edinburgh Pathway Notation scheme (mEPN) [39] and used to analyze the diurnal pattern of gene expression in the mouse kidney [40] using a stochastic Petri net-based approach [41]. The model encapsulates the interactions between 69 molecular species and contains 2,013 components and 2,100 interactions. All pathway components are labelled using standard nomenclature (HGNC gene id), and any modifications to those
components are explicitly stated in their labels. Moreover, proteins, genes and biochemicals
are hyperlinked to online resources, e.g., NCBI gene, ChEMBL and interactions between
components (process nodes) are annotated with publications providing supporting evidence.
In this respect, models can also serve as descriptive diagrams of known events that can be
easily evaluated and reused.

Reinforcing this idea, Paul Thomas introduced *Gene Ontology Activity Modeling*. Gene Ontology (GO) annotations are the most comprehensive structured representation of gene function and are widely used in the interpretation of genome-wide experimental data. However, because an individual GO annotation associates a single gene product with a single GO term, it is only a partial description of the gene function, which limits the expressiveness of annotations and their application in computational analysis of experimental data. To address this limitation, Thomas *et al*. have developed a novel framework, GO Causal Activity Modeling (GO-CAM), for linking multiple GO annotations into an integrated model of a biological system. GO-CAM supports modelling at multiple levels, from individual gene products to complex regulatory and metabolic pathways, and can be applied in network analysis and systems biology modelling, or converted into standard GO annotations for traditional GO-based analyses. Paul Thomas further presented the Noctua Modeling Tool used by GO Consortium curators to create GO-CAM models, from existing GO annotations or from scratch.

Finally, Anna Niarakis (UEVE, University of Paris-Saclay) closed the session by introducing
the *automated inference of annotated Boolean models from molecular interaction maps using
CaSQ*. She proposed a methodology to convert complex molecular maps into computable
logical models. Molecular interaction maps have emerged as a useful way of representing
biological mechanisms, based on information mining and human curation [40]. Nevertheless, their static nature does not allow for *in silico* simulations. With Sylvain Soliman (INRIA, Paris-Saclay), they have developed CaSQ [42], a tool that infers preliminary Boolean rules based on the topology and semantics of the molecular interaction maps, transforming these maps to executable Boolean models. They used a state-of-the-art molecular interaction map for Rheumatoid Arthritis [43,44] as a case study, but the tool can handle various maps differing in size and complexity and supports the SBGN standard. CaSQ inferred models are encoded in SBML *qual*, while references, annotations and layout are retained, thereby facilitating interoperability and model reusability.

## Tools and modelling platforms for dynamical analysis of logical models

The afternoon sessions highlighted the efforts of the community to develop methodologies and software that address issues of interoperability, reproducibility and reusability of modelling efforts. The level of annotation and the amount of curation are highly dependent on the modeller and on the capabilities of the existing tools to support this type of information in both human and machine-readable formats.

Tomáš Helikar (University of Nebraska-Lincoln) introduced *Cell Collective - Enabling accessible and collaborative construction and analysis of comprehensive and annotated models*. Cell Collective is a computational modelling platform for the collaborative construction, simulation, and analyses of large-scale dynamic (logical) models of biological and biochemical processes [3,4,45]. It contains nearly 100 public, peer-reviewed logical models of various biological and biochemical processes. To ease the reuse and expansion of existing models, every component and interaction is annotated to track the biological data
used to build the model. Models in Cell Collective can be created either *de novo* or imported using SBML-*qual*. Models are accessible in Cell Collective, where they can be simulated and further developed, or can be downloaded in SBML *qual* format, including via its public API [46].

Gaultier Stoll (Centre de Recherche des Cordeliers, INSERM) and Vincent Noël (Institut Curie) presented *MaBoSS* (*Markovian Boolean Stochastic Simulator ecosystem*). *MaBoSS* is a tool for simulating logical models with continuous-time Markov processes [47]. Stochastic simulations allow the computation of the probabilities of each state of the model over time. Over the years, *MaBoSS* was extended [48] and various tools were developed, including *UPMaBoSS*, enabling the study the dynamical behaviour of cell populations (including its size), and *PhysiBoSS*, based on an agent-based formalism where each agent is a logical model run with *MaBoSS*. A model of cell fate decision was used to showcase different ways of running the tools: through the command line, through the *CoLoMoTo* Jupyter interactive notebook, showing the interoperability of the tool, and using the python library *pymaboss* [49].

Vasundra Touré (DrugLogics group, NTNU) presented *The Minimum Information about a Molecular Interaction Causal Statement (MI2CAST): a set of guidelines for the annotation of molecular causal interactions* [23]. The NTNU group proposes MI2CAST as a standard for representing causal statements that can serve as a checklist that can be followed in curation processes for capturing the essential contextual information about a causal relationship, to ensure clarity, uniformity and reusability of the data across resources. MI2CAST has been developed in collaboration with the International Molecular Exchange (IMEx) consortium [50] and Human Proteome Organization - Proteomics Standards Initiative (HUPO-PSI) [51].
The NTNU group has also implemented the MI2CAST guidelines and annotation terms in a
prototype curation tool based on the VSM foundation [21], named causalBuilder [52].

Julio Saez-Rodriguez (Heidelberg University) focused on *Integrating knowledge and experimental data to build context-specific logic models*. The general pipeline involves obtaining existing prior knowledge on pathways from available public resources using OmniPath [52], building a logic model from this prior knowledge, and training it to data with tools such as *CellNOpt* (for targeted readouts [53]), *PHONEMeS* (for untargeted mass spectrometry [54]), and *CARNIVAL* (for gene expression, [54]). Regarding annotations, OmniPath provides information about localisation, function, disease relationships, proteins and complexes based on 36 resources. Collectively, OmniPath provides 2,200,000 annotation entries for 20,000 human proteins and 16,500 complexes and is available via a Python module, an R package, as a web service, or from Cytoscape [55,56].

Aurelien Naldi (Ecole Normale Superieure, Paris) presented *The CoLoMoTo Interactive Notebook*, which provides a unified environment to edit, execute, share, and reproduce analyses of Boolean and multi-valued models of biological networks. This framework combines the power of different software tools to ensure reproducibility and to reduce their learning curve. The CoLoMoTo Interactive Notebook currently eases access to GINsim, BioLQM [57], Pint [58], MaBoSS, and Cell Collective. More tools will be included in the future. Computational workflows can be edited through a web interface based on the Jupyter notebook, enabling the inclusion of textual annotations, along with the explicit code to execute, as well as the visualisation of the results. The framework is distributed as a Docker image with the tools ready to use without any installation step besides Docker, which can run on Linux, macOS, and Microsoft Windows systems.
Lastly, Eugenia Oshurko (Ecole Normale Supérieure, Lyon) presented *KAMISudio: an
environment for biocuration of cellular signalling knowledge* [59] suitable for rule-based
modelling languages, such as Kappa [60] and BioNetGet [61]. KAMISudio environment is
based on the KAMI biocuration framework that aims to decouple knowledge curation from
model building [62]. The main features of the KAMISudio environment can be used for
semi-automatic curation of large corpora of cellular signalling knowledge and for dynamic
study of the modelled systems.

Round table discussion

The general discussion highlighted four important aspects, namely (1) the need to provide
annotated models that would include textual annotations, bibliographic references and
crosslinks to knowledge resources through the use of common identifiers, (2) the importance
of creating interfaces for automatic integration of annotations by leveraging the wealth of
curated interactions in dedicated databases, (3) the utility of agreeing on best practices, use of
standards and on the minimum information required to ensure model reproducibility and
reusability, and lastly (4) the use of common repositories for logical models that would foster
interactions and facilitate exchanges between scientists interested in reusing models. The
need to encourage novel publications with logical models to be systematically submitted to
one of the model repositories was also discussed, as this would increase visibility, ease
reproducibility, and promote reusability of logical models.

Roadmap to best practices for the Curation and Annotation of Logical
Models (CALM) in biology
Based on these discussions, four interdependent milestones were identified for the roadmap to curation and annotation of logical models in biology (Figure 2):

a) The first milestone concerns the **reproducibility of the analyses of discrete models**.

The use of common, standardised formats (e.g., SBML packages *qual*, *layout*, *render*, etc.) would greatly facilitate the interoperability between different tools and the development of integrative pipelines. For example, the CoLoMoTo notebook could be expanded to include more tools, offering a flexible way of performing dynamical analyses in a fully transparent and reproducible manner. To achieve this goal, the logical modelling community aims to work close with the communities developing standards, such as SBML, the Simulation Experiment Description Markup Language (SED-ML) and Computational Modelling in Biology Network (COMBINE) to contribute to community efforts and make sure that the standards developed are in line with the specificities of the logical formalism.

b) The second milestone concerns the **minimum information for annotating a model, and also new mechanisms to encode such information in SBML-qual**. The information should be stored in human and machine-readable form, for example, by using Resource Description Framework (RDF) tags [63]. SBML format also provides the possibility to associate Systems Biology Ontology (SBO) terms outside of RDFs; however, unified storage of all model annotations in RDF could provide a simple, yet an efficient standard way of annotating logical models. Supported by larger computational modelling communities (e.g., COMBINE), RDF is considered the *de facto* standard for encoding annotations [64]. The community should discuss and agree on the best way of integrating annotations in SBML-qual, notably which tags and which SBML elements to use, while also leveraging the experience of the SBML
community and BioModels curation practices. Notably, the SBML specification documents [7] already propose a systematic way of annotation that can be adapted to logical models. Additionally, the logical modelling community should define specific needs that are not covered yet by existing standards (i.e., MIRIAM identifiers and BioModels.net qualifiers [65]) and propose feasible solutions. The minimum information for annotation could be proposed as a prerequisite for publishing a logical model in peer-reviewed journals. Table 2 lists suggested minimum qualifiers that could be used in order to annotate a model, in line with MIRIAM and BioModels suggestions. Furthermore, to aid model developers and curators, new tools need to be developed to aid the enrichment of models with as many relevant and useful annotations as possible. The metadata information for one of the three curated logical models currently available in BioModels and the corresponding code block of the xml file are exemplified in Figure 3. While the logical modelling community has made progress towards identifying important aspects of annotations, much work remains to be done. For example, the community is currently discussing the appropriate “depth” of annotations for each logical function. For example, does each variable and operator between variables in a logical function need to be annotated (such as currently available in Cell Collective)? While this level of annotations can add to the work-load of the modeller/curator, one might argue that providing citable experimental evidence for such aspects for the regulatory mechanism of each component will only increase the transparency of the model. The qual:transition component in the SBML model could be proposed for the annotation of causal interaction, however, this choice (already employed by some tools i.e., CaSQ, Cell Collective) raises issues concerning the cases where a more precise annotation would be needed.
c) The third milestone refers to the **collaboration between modellers and curators to bridge the gap between storing information and reusing information.** Automated procedures for model annotation and enrichment could further help to maintain models up to date. Keeping track of literature information used to derive logical formulae can further foster model accuracy and enhance reusability. To make steps forward, the logical modelling community aims to work closely with biocurators and knowledge platform developers to identify best practices. An obvious way would be to agree on the use of common and well-established identifiers like UniProt, GO, HGCN, SBO that would allow unambiguous identification of a model component with simultaneous access to the knowledge resource through crosslinks. This direct linking of model annotations to curated knowledge sources via standard identifiers could help significantly in establishing quality control checks regarding annotation and biological content.

d) The last milestone concerns **fully leveraging available model repositories**. Several logical model repositories exist, including Cell Collective, GINsim and PyBoolNet [66]. The Cell Collective provides models in several formats, including SBML *qual*. The GINsim's model repository provides models in the GINML format, which can be converted to SBML *qual* (and other formats) using GINsim and BioLQM. Simultaneously, BioModels is one of the largest repositories of mathematical, SBML-encoded models. However, it has been traditionally focused on models described with other mathematical frameworks, and lacks processes to curate logical models. Indeed, the logical modelling community has started to work closely with BioModels team to set up best practices and model quality checks that will be applicable to logical models. The aim is to create a dedicated collection of logical
models within BioModels, which would provide an additional resource with curated
logical models. In Box 1, we show a curated logical model stored in BioModels
(BIOMD0000000593) annotated as a sample case.

The logical modelling community should also decide if the suggestions of the COMBINE
community, as stated in Neal et al. [64], regarding the storage of annotations in a separate file
could be adopted. While this would allow for more flexibility in terms of knowledge
resources' choices for model annotation, i.e., one model file with several annotation files with
different sources, it would add the extra burden of file synchronization. However,
dissociating model from model annotation could be in line with the approaches and
methodologies presented in the first session of the meeting regarding the separation of the
biocuration from the model layout and refinement. An additional point to consider is the
simulation settings and their specifications through an established standard such as SED-ML
[67,68], which will likely require some adaptation to suit logical model simulations. In this
respect, the COMBINE Archive format could offer a possible solution, as it provides a
standardised way to bundle this type of files together [64].

Key Points

* The identified milestones will help the community of logical modelling to coordinate efforts for reproducible research.

* Standards for minimum curation will help unify formats and annotations, in an effort to provide models of better accuracy and quality.

* Transparency in curation and standardised annotations will enhance model reusability.

* Format harmonisation will facilitate interoperability and integration of existing tools in seamless pipelines.
* Collaboration between modellers and curators will foster model enrichment and updating, taking advantage of the wealth of information stored in databases and knowledge bases.

* The use of a common repository will reinforce quality protocols and checks for models, which could even be used prior to publication.

## Outcomes and Outlook

The [BC]2 workshop on annotation and curation of logical models in biology brought together people from different communities, such as biocurators, modellers, methodology and software developers. The round table discussion clarified common objectives together with milestones on the roadmap to best practices. Presentations and discussions highlighted efforts and resources that can be used for enhancing reproducibility and model contextualisation. The authors have started to form working groups and will continue to foster communication and exchanges first among the logical modelling community and also by reaching out to other communities with similar interests, to attain these collective goals.

<table><caption>**Table 2: Suggestion of minimum qualifiers for the annotation of logical models.** The hasState qualifier could be added to account for a node's state (qualitative levels).</caption><thead><tr><th>Model annotation levels</th><th>Minimum Qualifiers</th><th>Examples of knowledge sources stored in RDFs</th></tr></thead><tbody><tr><td>Model</td><td><strong>Model Qualifiers: bqmodel</strong><br><strong>is, identity</strong><br>This qualifier might be used to link an encoded model to a database of models.<br><br><strong>isDescribedBy, description</strong><br>This relation might be used to link a model to the</td><td>PMID, BioModels ID, doi, CC ID, GINsim ID, GO</td></tr></tbody></table>
<table><tr><td></td><td>literature that describes it.<br/><b>hasTaxon, taxon</b><br/>This qualifier might be used to indicate taxonomy/organism (i.e: human, plant, animal).<br/><b>isVersionOf, version</b><br/>This qualifier can be used to link a model to the Gene Ontology terms regarding the biological function described.<br/><b>hasProperty, property</b><br/>This relation could be used to indicate mathematical formalism.<br/><b>isDerivedFrom, origin</b><br/>This relation may be used to express a refinement or adaptation in usage for a previously described model</td><td></td></tr><tr><td><b>Qualitative Species</b></td><td><b>Biology Qualifiers: bqbiol</b><br/><b>is, identity</b><br/>This relation might be used to link a biological entity to its exact counterpart in a database.<br/><b>isDescribedBy, description</b><br/>This relation should be used to link a species to the literature that describes the role of that species or its presence in the system of interest.<br/><b>hasVersion, version</b><br/>This relation may be used to represent an isoform or modified form of a biological entity.<br/><b>hasState, state</b><br/>This relation could be used to describe the state of a biological entity.</td><td>GO, UniProt, HGCN, PMID</td></tr><tr><td><b>Causal interactions/transitions</b></td><td><b>Biology Qualifiers: bqbiol</b><br/><b>hasProperty, property</b><br/>This relation might be used when a biological entity exhibits a certain enzymatic activity or exerts a specific function.<br/><b>isDescribedBy, description</b><br/>This relation should be used, for instance, to link a reaction to the literature that describes it.</td><td>KEGG, REACTOME, PMID</td></tr></table>

The complete list of abstracts can be found in the supplementary material *Abstract_Booklet*.
## Authors' information

**Anna Niarakis** is an Associate Professor at Univ Evry, University of Paris-Saclay. Her research focuses on the application of Computational Systems Biology approaches in human diseases, including the construction of disease maps, tool development for model inference, network integration and dynamical modelling.

**Martin Kuiper** is Professor in Systems Biology at the Department of Biology of the Norwegian University of Science and Technology. His group develops enabling technology for data analysis and knowledge management, among others for the construction of logical models for systems medicine.

**Marek Ostaszewski** is a scientist and a project manager at the LCSB, working on knowledge management and visualization in systems biomedicine, in particular in Parkinson's disease, including clinical research.

**Rahuman S Malik Sheriff** is a project leader at the European Bioinformatics Institute (EMBL-EBI). He leads the development of BioModels infrastructure as well as curation and annotation of computational models in the standard formats. He is one of the members of the SBML Editorial Board. His interest includes building tools and resources for system biology modelling.

**Cristina Casals-Casas** is a biocurator in the Swiss-Prot group at SIB. Her main responsibilities include the literature-based expert curation of mammalian proteins for UniProtKB and GO knowledge bases. She is also involved in research projects that require expert biocuration.

**Denis Thieffry** is currently Professor of Systems Biology at the Ecole Normale Supérieure (ENS), Paris, France, and part-time Invited Research Professor at the Cancer Science Institute of the National University of Singapore.
**Tom C. Freeman** is Professor holding the chair of Systems Immunology at the University of Edinburgh. He has a broad range of interests ranging from immune cell differentiation, transcriptomics, data visualisation, and pathway modelling and simulation.

**Paul Thomas** is an Associate Professor at the University of Southern California. His research is in the area of computational analysis of genomic data, with an emphasis on gene function and evolution. In addition to founding and continuing development on the PANTHER project, Dr. Thomas is a director of the Gene Ontology Consortium, one of the largest and best-known bioinformatics projects in the world.

**Vasundra Touré** is a PhD student at the Norwegian University of Science and Technology. She is interested in the knowledge management and data extraction of molecular causal statements.

**Vincent Noël** is a postdoctoral fellow currently working in Institut Curie. His research interests concern modelling of biological systems, from the construction and the analysis of models to the optimization of their simulations.

**Julio Saez-Rodriguez** is Professor of Medical Bioinformatics and Data Analysis at the Faculty of Medicine of Heidelberg University, director of the Institute for Computational Biomedicine, group leader of the EMBL-Heidelberg University Molecular Medicine Partnership Unit and a co-director of the DREAM challenges. His research focuses on computational methods to understand and treat the deregulation of cellular networks in disease.

**Aurélien Naldi** is a postdoctoral fellow currently working in the Ecole Normale Supérieure (ENS). His interests cover the integration of advanced computational methods into software tools for the analysis of complex biological models.

**Eugenia Oshurko** is a PhD student at ENS Lyon. Her research interests include graph rewriting, graph-based databases, and knowledge representation and their application to the curation of biological knowledge relevant to building executable rule-based models.
**Ioannis Xenarios** is an Associate Professor of Computational Biology at the CHUV/UNIL/LICR. His research aims to model the tumor microenvironment of T cells infiltrates (TIL) and monocytes. He develops both curated and data driven based model to characterize and provide experimentally testable hypothesis using drugs and immunotherapies.

**Sylvain Soliman** is a researcher at Inria, Saclay. His research interests focus around Computational Biology and Theoretical Computer Science. He has been one of the main developers and maintainers of the BIOCHAM modelling and analysis platform for more than ten years.

**Claudine Chaouiya** is an Associate Professor at I2M, Aix-Marseille University. Her research focuses on developing appropriate methods for the analysis of large logical models of regulatory and signalling networks. Methodological advancements and tool development are motivated and further validated through modelling projects revolving around cell fate decisions, in single and multi-cellular contexts.

**Tomas Helikar** is an Associate Professor in the Department of Biochemistry at the University of Nebraska-Lincoln. His research focuses on the use of integrative multi-approach modelling pipelines for dynamical analysis of biological networks. More precisely, his studies focus on understanding how aberrant changes in biological networks result in disease so that we could strategically develop more effective therapies.

**Laurence Calzone** is a researcher in Institut Curie. Her research activities revolve around building knowledge maps, constructing logical models, and developing tools for improving and optimizing the qualitative models and their predictions.

## Acknowledgements

The authors would like to thank the SysMod community and the [BC]2 organisers for supporting and hosting the meeting/workshop in Basel.
**Funding**

Not applicable

**Competing Interests**

The authors declare no competing interests regarding the content of this manuscript

**References**

1. Allison DB, Shiffrin RM, Stodden V. Reproducibility of research: Issues and proposed remedies. Proc. Natl. Acad. Sci. 2018; 115:2561–2562

2. Naldi A, Hernandez C, Abou-Jaoudé W, et al. Logical Modeling and Analysis of Cellular Regulatory Networks With GINsim 3.0. Front. Physiol. 2018; 9:646

3. Helikar T, Kowal B, McClenathan S, et al. The Cell Collective: Toward an open and collaborative approach to systems biology. BMC Syst. Biol. 2012; 6:96

4. Helikar T, Kowal B, Rogers JA. A cell simulator platform: the cell collective. Clin. Pharmacol. Ther. 2013; 93:393–395

5. Malik-Sheriff RS, Glont M, Nguyen TVN, et al. BioModels—15 years of sharing computational models in life science. Nucleic Acids Res. 2019;

6. Hucka M, Finney A, Sauro HM, et al. The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models. Bioinformatics 2003; 19:524–531

7. Chaouiya C, Keating SM, Berenguier D, et al. SBML Level 3 package: Qualitative Models, Version 1, Release 1. J. Integr. Bioinforma. 2015; 12:

8. Chaouiya C, Bérenguier D, Keating SM, et al. SBML qualitative models: a model representation format and infrastructure to foster interactions between qualitative modelling formalisms and tools. BMC Syst. Biol. 2013; 7:135

9. Naldi A, Monteiro PT, Müssel C, et al. Cooperative development of logical modelling standards and tools with CoLoMoTo. Bioinforma. Oxf. Engl. 2015; 31:1154–1159

10. SysMod. SysMod Community of Special Interest.

11. BASEL LIFE 2019, 9–12 September 2019, Congress Center Basel, Switzerland.

12. BaselLife 2019 Workshops and tutorials.

13. BioKB. https://biokb.lcsb.uni.lu/ 2019;

14. Gawron P, Ostaszewski M, Satagopam V, et al. MINERVA-a platform for visualization and curation of molecular interaction networks. NPJ Syst. Biol. Appl. 2016; 2:16020

15. Bateman A, Martin MJ, O’Donovan C, et al. UniProt: the universal protein knowledgebase. Nucleic Acids Res. 2017; 45:D158–D169

16. Perfetto L, Briganti L, Calderone A, et al. SIGNOR: a database of causal relationships between biological entities. Nucleic Acids Res. 2016; 44:D548–D554
17. Braschi B, Denny P, Gray K, et al. Genenames.org: the HGNC and VGNC resources in 2019. Nucleic Acids Res. 2019; 47:D786–D792

18. The Gene Ontology Consortium. The Gene Ontology Resource: 20 years and still GOing strong. Nucleic Acids Res. 2019; 47:D330–D338

19. Fabregat A, Jupe S, Matthews L, et al. The Reactome Pathway Knowledgebase. Nucleic Acids Res. 2018; 46:D649–D655

20. The Druglogics Initiative.

21. vsmjs. https://github.com/vsmjs. GitHub 2019;

22. SciCura. http://demo.scicura.org/. 2019;

23. MI2CAST. https://github.com/MI2CAST 2019;

24. Biryukov M, Groues V, Satagopam V. BioKB - Text Mining and Semantic Technologies for the Biomedical Content Discovery. 2019; 12

25. Hoksza D, Gawron P, Ostaszewski M, et al. MINERVA API and plugins: opening molecular network analysis and visualization to the community. Bioinformatics 2019; 35:4496–4498

26. Hoksza D, Gawron P, Ostaszewski M, et al. Closing the gap between formats for storing layout information in systems biology. Brief. Bioinform. 2019;

27. Kitano H, Funahashi A, Matsuoka Y, et al. Using process diagrams for the graphical representation of biological networks. Nat. Biotechnol. 2005; 23:961–966

28. Newt Editor. http://newteditor.org/index.html 2019;

29. Glont M, Nguyen TVN, Graesslin M, et al. BioModels: expanding horizons to include more modelling approaches and formats. Nucleic Acids Res. 2018; D1248–D1253

30. Le Novère N, Finney A, Hucka M, et al. Minimum information requested in the annotation of biochemical models (MIRIAM). Nat. Biotechnol. 2005; 23:1509–1515

31. Yates AD, Achuthan P, Akanni W, et al. Ensembl 2020. Nucleic Acids Res. 2019;

32. Federhen S. The NCBI Taxonomy database. Nucleic Acids Res. 2012; D136–D143

33. Degtyarenko K, de Matos P, Ennis M, et al. ChEBI: a database and ontology for chemical entities of biological interest. Nucleic Acids Res. 2008; 36:D344–D350

34. Mathematical Modelling Ontology. https://www.ebi.ac.uk/ols/ontologies/mamo. 2019;

35. Systems Biology Ontology.

36. BRENDA tissue / enzyme source. https://www.ebi.ac.uk/ols/ontologies/bto. 2019;

37. Bekkar A, Estreicher A, Niknejad A, et al. Expert curation for building network-based dynamical models: a case study on atherosclerotic plaque formation. Database J. Biol. Databases Curation 2018; 2018:

38. Naldi A, Hernandez C, Levy N, et al. The CoLoMoTo Interactive Notebook: Accessible and Reproducible Computational Analyses for Qualitative Biological Networks. Front. Physiol. 2018; 9:

39. Freeman TC, Raza S, Theocharidis A, et al. The mEPN scheme: an intuitive and flexible graphical system for rendering biological pathways. BMC Syst. Biol. 2010; 4:65

40. Ivy JR, Shih B, Hogenesch JB, et al. A detailed graphical and computational model of the mammalian renal circadian clock. bioRxiv 2019; 795906

41. Livigni A, O’Hara L, Polak ME, et al. Petri Net-Based Graphical and Computational Modelling of Biological Systems. bioRxiv 2016; 047043

42. Aghamiri, S.S, Singh, V, Naldi A, et al. Automated inference of Boolean models
from molecular interaction maps using CaSQ. *Bioinformatics* Under review.;

43. Singh V, Ostaszewski M, Kalliolias GD, et al. Computational Systems Biology Approach for the Study of Rheumatoid Arthritis: From a Molecular Map to a Dynamical Model. *Genomics Comput. Biol.* 2018; 4:

44. Singh V, Kalliolias GD, Ostaszewski M, et al. RA-map: Building a state-of-the-art interactive knowledge base for rheumatoid arthritis. *Database Under Review*;

45. Helikar T, Cutucache CE, Dahlquist LM, et al. Integrating interactive computational modeling in biology curricula. *PLoS Comput. Biol.* 2015; 11:e1004131

46. Kowal BM, Schreier TR, Dauer JT, et al. Programmatic access to logical models in the Cell Collective modeling environment via a REST API. *Biosystems* 2016; 139:12–16

47. Stoll G, Caron B, Viara E, et al. MaBoSS 2.0: an environment for stochastic Boolean modeling. *Bioinformatics* 2017; 33:2226–2228

48. Letort G, Montagud A, Stoll G, et al. PhysiBoSS: a multi-scale agent-based modelling framework integrating physical dimension and cell signalling. *Bioinformatics* 2019; 35:1188–1196

49. . pyMaBoSS — pyMaBoSS 0.3.2 documentation.

50. Orchard S, Kerrien S, Abbani S, et al. Protein interaction data curation: the International Molecular Exchange (IMEx) consortium. *Nat. Methods* 2012; 9:345–350

51. Taylor CF, Hermjakob H, Julian RK, et al. The work of the Human Proteome Organisation's Proteomics Standards Initiative (HUPO PSI). *Omics J. Integr. Biol.* 2006; 10:145–151

52. causalBuilder. https://vtoure.github.io/causalBuilder//causalBuilder/. causalBuilder 2019;

53. Terfve CDA, Wilkes EH, Casado P, et al. Large-scale models of signal propagation in human cells derived from discovery phosphoproteomic data. *Nat. Commun.* 2015; 6:1–11

54. Liu A, Trairatphisan P, Gjerga E, et al. From expression footprints to causal pathways: contextualizing large signaling networks with CARNIVAL. *Npj Syst. Biol. Appl.* 2019; 5:1–10

55. Türei D, Korcsmáros T, Saez-Rodriguez J. OmniPath: guidelines and gateway for literature-curated signaling pathway resources. *Nat. Methods* 2016; 13:966–967

56. Ceccarelli F, Turei D, Gabor A, et al. Bringing data from curated pathway resources to Cytoscape with OmniPath. *Bioinformatics*

57. Naldi A. BioLQM: A Java Toolkit for the Manipulation and Conversion of Logical Qualitative Models of Biological Networks. *Front. Physiol.* 2018; 9:

58. Paulevé L. Pint: A Static Analyzer for Transient Dynamics of Qualitative Networks with IPython Interface. 2017;

59. KAMISudio. http://kamistudio.ens-lyon.fr/about. 2019;

60. Boutillier P, Maasha M, Li X, et al. The Kappa platform for rule-based modeling. *Bioinformatics* 2018; 34:i583–i592

61. Faeder JR, Blinov ML, Hlavacek WS. Rule-Based Modeling of Biochemical Systems with BioNetGen. *Syst. Biol.* 2009; 113–167

62. Kappa-Dev/KAMI. https://github.com/Kappa-Dev/KAMI. 2019;

63. Novère NL, Finney A. A simple scheme for annotating SBML with references to controlled vocabularies and database entries. 2005; 13

64. Neal ML, König M, Nickerson D, et al. Harmonizing semantic annotations for computational models in biology. *Brief. Bioinform.* 2019; 20:540–550
65. BioModels.net Qualifiers | COMBINE.

http://co.mbine.org/specifications/qualifiers. 2019;

66. Klarner H, Streck A, Siebert H. PyBoolNet: a python package for the generation, analysis and visualization of boolean networks. Bioinforma. Oxf. Engl. 2017; 33:770-772

67. Bergmann FT, Cooper J, König M, et al. Simulation Experiment Description Markup Language (SED-ML) Level 1 Version 3 (L1V3). J. Integr. Bioinform. 2018; 15:

68. Waltemath D, Adams R, Bergmann FT, et al. Reproducible computational biology experiments with SED-ML - The Simulation Experiment Description Markup Language. BMC Syst. Biol. 2011; 5:198

<figure><img src="image_1.png" /><figcaption>**Figure 1.** Four main thematic axes of the presentations and the round table discussion of the meeting. Biocuration platforms, available model repositories, tool development and integrative methodologies were the main subjects of the meeting. All presentations highlighted the need for standards in model annotation and curation.</figcaption></figure>
<figure><img src="image_2.png" /><figcaption>**Figure 2. Roadmap to Curation and Annotation of Logical Models in Biology.** Four milestones were identified as key steps in the roadmap to best practices for logical models annotation and curation: integrated pipelines for reproducible research, standards for SBML qual annotations, automation of models enrichment and the use of a common repository.</figcaption></figure>
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /><figcaption>**Figure 3.** A logical model in Biomodels database. Metadata information for the curated logical model in BioModels database (upper panel) and the corresponding block code (lower panel).</figcaption></figure>
### Use case of a curated and annotated logic model (BIOMD0000000593) in
BioModels.

* Cross-references to well-established ontologies like GO, UNIPROT, NICT, SBO etc. are added to NODES (SPECIES) and CAUSAL-INTERACTIONS using RDF.

* Use of COMBINE qualifiers http://co.mbine.org/standards/qualifiers where possible.

Annotations are added at two levels:

1. Model Level annotation

```xml
<bqmodel:isDerivedFrom>
  <rdf:Bag>
    <rdf:li rdf:resource="http://identifiers.org/pubmed/12871957"/>
    <rdf:li rdf:resource="http://identifiers.org/pubmed/16314431"/>
  </rdf:Bag>
</bqmodel:isDerivedFrom>
```

2. Model component level annotation:

a. NODE (SPECIES)

```xml
<qual:qualitativeSpecies metaid="species11" qual:compartment="default"
    qual:constant="false" qual:id="IL21" qual:maxLevel="1" qual:name="IL21">
  <annotation>
    <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
      xmlns:dc="http://purl.org/dc/elements/1.1/"
      xmlns:dcterms="http://purl.org/dc/terms/"
      xmlns:bqbiol="http://biomodels.net/biology-qualifiers/">
      <rdf:Description rdf:about="#species11">
        <bqbiol:is>
          <rdf:Bag>
            <rdf:li rdf:resource="http://identifiers.org/uniprot/Q9HBE4"/>
          </rdf:Bag>
        </bqbiol:is>
      </rdf:Description>
    </rdf:RDF>
  </annotation>
</qual:qualitativeSpecies>
```
## 2. Model component level annotation:

### b. CAUSAL INTERACTIONS

```xml
<qual:transition metaid="rxn01" qual:id="tr_TBET" qual:name="Interactions targeting TBET">
  <qual:listOfInputs>
    <qual:input metaid="rxn01_input01" qual:qualitativeSpecies="IL4"
                 qual:transitionEffect="none"/>
  </qual:listOfInputs>
  <qual:listOfOutputs>
    <qual:output metaid="rxn01_output01" qual:qualitativeSpecies="TBET"
                 qual:transitionEffect="assignmentLevel"/>
  </qual:listOfOutputs>
  <qual:listOfFunctionTerms>
    <qual:functionTerm metaid="rxn01_function01" qual resultLevel="1">
      <math xmlns="http://www.w3.org/1998/Math/MathML">
        <apply>
          <not/>
          <apply>
            <or
              /> <!-- Empty operator -->
            <apply>
              <eq/> <!-- Empty operator -->
              <ci> IL4 </ci>
              <cn type="integer"> 1 </cn>
            </apply>
          </apply>
        </apply>
      </math>
    </qual:functionTerm>
  </qual:listOfFunctionTerms>
</qual:transition>
```

**Box 1. An example of annotating a logical model using RDFs.** BioModels propose a two level annotation, model and model component. Model components are in turn annotated in two levels: nodes and causal interactions. A color code is used to highlight the different code blocks that refer to each level of annotation. Code blocks are excerpts from a syntactically valid SBML *qual* file.
**Table 1** Summary of different topics and presentations.

**Table 2: Suggestion of minimum qualifiers for the annotation of logical models.** The `hasState` qualifier could be added to account for a node's state (qualitative levels).