<figure><img src="image_1.png" /></figure>

HAL
open science

# Narrowing the Gap between Open Standards Policy and Practice: The Dutch e-Government Experience

Rutger Lammers, Erwin Folmer, Michel Ehrenhard

<figure><img src="image_2.png" /></figure>

## To cite this version:

Rutger Lammers, Erwin Folmer, Michel Ehrenhard. Narrowing the Gap between Open Standards Policy and Practice: The Dutch e-Government Experience. Joint IFIP TC 8 and TC 6 International Conferences on E-Government, E-Services and Global Processes (EGES) / Global Information Systems Processes (GISP), Held as Part of World Computer Congress (WCC), Sep 2010, Brisbane, Australia. pp.7-18, 10.1007/978-3-642-15346-4_2. hal-01054636

HAL Id: hal-01054636

https://inria.hal.science/hal-01054636

Submitted on 7 Aug 2014

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Narrowing the Gap between Open Standards Policy and Practice: The Dutch e-Government Experience

Rutger Lammers,<fnref n="1" />,<fnref n="2" /> Erwin Folmer,<fnref n="1" />, <fnref n="2" />,<fnref n="3" /> Michel Ehrenhard<fnref n="2" />

**Abstract.** Interoperability in the public sector can be improved by the use of open standards. Nonetheless, the openness of standards in government policies is debatable. This paper introduces the Dutch government policy on open standards, and will introduce a multi-dimensional view (and model) on openness rather than a one-dimensional strict definition. Applicability of the multi-dimensional model is tested in a case study, which demonstrates that this model has value for standardization organizations active in the government domain. In future cases the model helps in understanding how government-related standardization organizations can influence openness in a situation-specific way and the model therefore narrows the gap between open standards policy and practice.

**Keywords:** standards, standardization, open, e-government, policy, interoperability

## 1 Introduction

High on the Dutch government's agenda is the creation of an innovative, competitive and enterprising economy. To achieve this goal the government has developed a policy in which it states that the governments needs to be a service-oriented and a client-friendly partner [1][2][11][15]. Important for the realization of this policy is that the IT infrastructure of the public sector is flexible [15]. To achieve flexibility, interoperability between governments and businesses, between governments and citizens and between government bodies themselves is an essential condition [2][15]. We define interoperability as the degree to which an information system is able to exchange information with other systems, in such a way that the meaning of information (semantics) does not change [2][15]. Interoperability provides a environment in which various systems can communicate with each other, not only syntactical, but also shared semantics. Therefore interoperability implies a common language between systems, which results in governmental supplier-independence, digital sustainability and transparency [11][15].

<footnote n="1">¹The Netherlands Open in Connection, The Hague, The Netherlands</footnote>

<footnote n="2">²University of Twente, Enschede, The Netherlands</footnote>

<footnote n="3">³TNO Information and Communication Technology, Enschede, The Netherlands
{rutger.lammers, erwin.folmer}@noiv.nl</footnote>
## Purpose of this paper

Within this paper open standards are seen as a mean to achieve the goal of interoperability. We will explain why a more sophisticated approach is needed than just a strict definition of what constitutes an open standard. Every situation is unique and a situation-specific process will result in a better fit of a standard. The purpose of this paper is to gain more insight in the openness of standards. Therefore, the central question of this paper is: "How can the openness of an open standard be influenced?" The unit of analysis in this report is a standard, but since the openness of a standard greatly depends on the process surrounding the standard, the framework may be generalized to the degree of openness of the standard developing organization (SDO) as a whole.

In this paper we answer this question by

1) Introducing the Dutch policy on open standards (chapter 2);

2) Defining a model on openness (chapter 3);

3) Operationalizing the model to make it useful for practice (chapter 4);

4) Using the operationalized model to analyze the openness of the StUF standard.

5) Discussing the results of this case study and drawing conclusions.

However, first the concepts of interoperability and open standards will be discussed briefly.

## Interoperability

Interoperability within the public sector can be achieved via various mechanisms. One way to achieve interoperability is by only using information systems from a single vendor [2]. These systems will most probably be able to perfectly exchange information with each other, by means of a proprietary interface between the systems. But, as the current IT portfolio of the public sector is very heterogeneous, and the sector is characterized by a plethora of different systems, this is not a realistic solution [11][15]. Moreover, single-vendor interoperability will not contribute to supplier-independence because the use of an proprietary interface by another vendor is usually a costly and complex process. Another mechanism to achieve interoperability is the use of open standards [11][18]. Standards are about collectively agreeing on the specifications for the interfaces between application, services, systems and networks that interact. Open standards differ from proprietary standards because participating in the process of developing, using and maintaining such a standard is in principle open to and freely accessible for everyone. The Dutch government has chosen to prefer open standards above proprietary standard, mainly to achieve interoperability within its IT architecture [1][2][11]. This paper focuses on standards for data exchange. These standards relate to technical and semantic interoperability and not so much to organizational interoperability [20].
## Open standards

'Open standard' is a broad term and in literature many definitions of an open standard are given [5][3][10]. There is a need for a more sophisticated approach to describing and examining open standards than just a one-dimensional and rigid definition. For the Dutch government it is essential to understand which parameters play an essential role in the openness of a standard. Understanding the parameters of openness contributes to the founded and deliberate selection and/or development of an open standard. The requirements to the parameters of openness will depend on the situation, since not every standard needs to be open on all aspects. For instance, the TCP/IP protocol does not need to have a very open change process, since most people are only interested in the open use of this standard. But there are also standards for which other aspects of openness are more important [5].

Standards are developed and maintained by a standard developing organization (SDO's). Many SDOs exist, ranging from formal standardization organizations such as ISO to fora such as W3C. The internal process of developing and maintaining a standard within these organizations varies. Naturally, the degree of openness of the process influences the degree of openness of a standard.

## 2 Dutch policy on open standards

To stimulate the use of open standards and to guide the change process, the Dutch government has initiated the action plan "The Netherlands Open in Connection" (Dutch abbreviation: NOiV), which consists of several action lines resulting in three objectives [1][11]:

1. Increase interoperability of information system in the Dutch public sector by accelerating the use of open standards

2. Reduce dependence on suppliers in the use of ICT through faster introduction of open standards

3. Promote a level playing field in the software market by using open standards

The Dutch government intends to encourage the use of open standards within the public and semi-public sector. The key focus is: use open standards, unless there is a very good reason why this is not possible, and indicate when open standards will indeed be implemented. This is the principle of 'comply or explain, and commit'. Through this principle the use of open standards will be given a firm foundation.

## Actors related to Dutch policy

Figure 1 gives more insight in the relations between organizations that are concerned with the execution of the Dutch policy on the use of open standards.
<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Actors related to Dutch policy</figcaption></figure>

## The Bureau Forum Standardization (BFS)

In order to be able to implement the government's policy, a process of selecting and applying open standards based on a clear framework for interoperability should be determined. This is done by the Bureau Forum Standardization (BFS) [2][12]: a program office that selects the open standards that fall under the rule 'comply or explain, and commit'. BFS selects standards for the 'comply or explain, and commit' list. BFS maintains two separate lists of open standards:

1. A list of 'comply-or-explain' standards. These standards are often not yet widely used within the Dutch public sector, but the use of these standards is mandatory. The standards on this list are usually semantic standards, like SETU (procurement), XBRL (finance) and StUF (administrative) and these standards are often a specific instance of a global standard (like SETU, which is a location-specific version of HR-XML).

2. A list of "frequently applied standards", standards which are widely used in the architecture of information systems. These 'defacto' standards are often technical, world-wide standards located at the lower layers in the OSI reference model, like the TCP/IP-protocol, SMTP et cetera. This list helps purchasers in the public sector in tendering among others.

## NOiV and action lines

The Dutch policy on the use of open standards consists of several action lines organized in the program NOiV [11]. Together these action lines contribute to the three objectives of the government's policy on open standards. Relevant action lines<fnref n="1" /> are discussed shortly.

Action line 1 – The Bureau Forum Standardization (BFS) publishes a basis list of open standards which is necessary for implementation of the 'comply or explain, and commit' principle [12][2].

<footnote n="1">¹The action plan also includes action lines on the adoption of open source software (action line 7-13). But, as this paper only considers open standards, these action lines will not be discussed further.</footnote>
Action line 2 – Institutions in the public and semi-public sector will introduce the 'comply or explain, and commit' principle for ICT orders. Public sector bodies and institutions are themselves responsible for the application of 'comply or explain, and commit', using self-policing measures.

* Comply: apply established open standards to orders for new IT systems or rebuilds and IT contract extension.

* Explain: exception criteria are: no open standard is available for the desired functionality; the open standard is not supported by multiple suppliers and on several platforms; conduct of business and/or service provision would be unacceptably jeopardized, including in terms of security or international agreements would be broken.

* Commit: give preference to the application of open standards so that an exception criterion is no longer applicable.

Action line 3 – BFS prepares a Dutch interoperability framework.

Action line 4 – IT orders can be voluntarily examined for advice.

Action line 5 – Enforcement via monitoring and ranking.

Action line 6 – Central governments support ODF.

In summary, this Dutch policy makes the use of open standards mandatory for all (semi) public organizations.

## 3 Open Standards definitions & models

Openness is not a black or white situation, and whatever definition is chosen, it is arbitrary. For example, take the criterion of the free or nominal fee for a standard, also discussed by Rada [17]. Imagine that we have to pay 20 euro for a standards specification. Some will call this an open and free specification (nominal price), others will say this does not satisfy the definition of open and free specification. And how will this change if the amount will be changed to 200 or 20.000 euro's? But all agree on that a standard for which 20 euro should be paid is less open than a standard that can be freely downloaded. Although open standards itself are without doubt, a lot of debate is going on about the definition of an open standard. We will not go into that discussion, simply because it distracts attention on how to achieve more open standards. The discussion is also the other way around: it focuses on creating a definition to which many as open perceived standards comply. The definition is adapted to current practice. In our opinion the definition of a standard should be more transparent and adaptable to a specific situation. However we argue that the definition is not that interesting when the government tries to improve the openness of standards. For that use a broad view on openness is needed instead of a strict and small definition of openness.
## Open standard definition used by Dutch governments

The definition of open standards which is used by Dutch government complies with the definition which is used by the European Commission as set by the IDABC program<fnref n="2" />) [7][11]:

* The standard is approved and will be maintained by a non-profit organization, and ongoing development will be on the basis of an open decision-making process that is accessible for all interested parties.

* The standards is published and the specification document for the standard is freely available or can be obtained for a nominal contribution. It must be possible for everyone to copy it, make it available and use it, free or for a nominal price

* The intellectual property – regarding any patents that may be present – of the standard or parts thereof is irrevocably made available on a royalty-free basis;

* There are no restrictions on reuse of the standard.

In addition to this definition, the government uses the following two specification in elaborating the action plan:

* Open Specification: an open specification is one that is published and whose specification document is freely available. Alternatively, it may be available for a nominal contribution. It must be possible for everyone to copy it, make it available and use it, free or for a nominal price

* Free Specification: a free specification is an open specification that is free of legal restrictions making its use and distribution difficult. The intellectual property – regarding any patents that may be present – of the standard or parts thereof is irrevocably made available on a royalty-free basis.

## Existing research on open standards

A scan of existing literature on open standards showed that in general there are very few models and definitions that exceed a rigid definition. There is a need for a more sophisticated in-depth model which describes the dimension on which a standards can be positioned. One very useful model on open standards comes from Krechmer [5]. Krechmer introduces ten requirements on open standards. These ten requirements function as dimensions in which a certain open standard can be positioned. This dimensioning helps in shifting the discussion on open standards from a dichotomous black or white situation to a more nuanced situation. This helps in achieving a better task technology fit as discussed in chapter one. However, it remains unclear how Krechmer derives these ten requirements, why there are ten and moreover, each requirements leaves a lot of freedom of interpretation. Another useful model comes from Andersen [4]. This work is related to the ten requirements described by Krechmer. Andersen made an operationalization on the openness of open standards based on desk research. Other definitions and models are introduced in table 1. The dimensions which are used in this table are derived from Krechmer. The work of

<footnote n="2">²IDABC stands for Interoperable Delivery of European eGovernment Services to public Administrations, Business and Citizens.</footnote>
Krechmer forms the theoretical basis of this research. Since each definition can be mapped to one or more requirements, the requirements of Krechmer is the most complete set of dimensions.

<table><caption>**Table 1.** Models and definitions on open standards</caption><thead><tr><th>Dimension (Krechmer) [5]</th><th>Andersen [4]</th><th>Perens [13]</th><th>NOiV [11]</th><th>IDABC [7]</th><th>Dimension (Krechmer) [5]</th><th>Andersen [4]</th><th>Perens [13]</th><th>NOiV [11]</th><th>IDABC [7]</th></tr></thead><tbody><tr><td>Open Meeting</td><td>x</td><td>x</td><td>x</td><td>x</td><td>Open Change</td><td>x</td><td>x</td><td></td><td></td></tr><tr><td>Consensus</td><td>x</td><td>x</td><td>x</td><td>x</td><td>Open Documents</td><td>x</td><td>x</td><td>x</td><td>x</td></tr><tr><td>Due Process</td><td>x</td><td>x</td><td></td><td></td><td>Open Interface</td><td></td><td>x</td><td></td><td></td></tr><tr><td>Open IPR</td><td>x</td><td>x</td><td>x</td><td>x</td><td>Open Access</td><td>x</td><td>x</td><td>x</td><td>x</td></tr><tr><td>One World</td><td>x</td><td></td><td></td><td></td><td>On-going Support</td><td>x</td><td></td><td></td><td></td></tr></tbody></table>

As stated earlier, the current quantity of academic literature on dimensions of open standards is low. When we compare the available literature, the work of Krechmer is the most complete and most cited work. Therefore the ten requirements are already very useful in practice and they will therefore function as the basis of this paper.

## 4 Operationalization of the model on openness

Although defined dimensions are needed, they will become even more valuable when they are operationalized to (measurable) consequences in practice. As shown in table 1, the dimensions which are used by Krechmer to analyze the openness of standards have a broader view than others. Therefore, the operationalization of openness will be based on these ten dimensions (requirements in terms of Krechmer). These ten dimensions can be interpreted as a must-have to be labeled as an open standard and are therefore very useful in determining at a first glance to which degree an open standard is really open. However, if we interpret the ten requirements as ten dimensions on which a standard can be positioned, we need more than just the dimensions. For example, it could be discussed what exactly an open meeting is. Is everybody free to join meetings? How are invitations for these meetings arranged? So, to further improve the usability of the ten requirements on open standards we will make a first step in the operationalization of the requirements, which results in an model useful in practice.

### Method

To identify variables for the ten dimensions on open standards a workshop was organized and also the work of Andersen [4] was used. The first part of the workshop had the character of an informal brainstorm session in which five experts on open standards were invited to come up with variables for the ten dimensions on open standards. These experts have a background in standardization, varying from developers, implementers and users.
The ten dimensions were discussed step-wise, following the format of “if you think of this requirement, what comes up to measure it?” The second part of the workshop consisted of a clustering of the results from the workshop, following the structure of the ten dimensions. During the brainstorm and clustering we followed Langley [19]. This resulted in three to five variables per requirement, which are considered valid by the experts. The results of the workshop were combined with existing literature on openness of standards, especially the work of Andersen. This synthesis resulted in the model on openness of standards.

<figure><img src="image_5.png" /><figcaption>Fig. 2. Process used in this research paper</figcaption></figure>

## Results

The synthesis of both workshop results and literature resulted in concrete variables for the ten dimensions on open standards. The result is a method, which is presented in Table 2. The ten dimensions of Table 1 are operationalized in one or more variables (some dimensions could not be operationalized further into two or more variables).

Each variable can be scored, using a three-point scale:

* 0: in documentation related to the standard, nothing can be found regarding this variable or it is explicitly stated no action related to this variable is present.

* 1: documentation related to the standard touches upon this variable in a positive way, but not fully compliant to the line of reasoning behind the variable. Improvement is possible.

* 2: documentation related to the standard explicitly touches upon the line of reasoning behind the variable. Improvement is not necessary.

When using this model to score a standard, the most open source of information will be official documentation by the SDO on a website. So, if information regarding a variable cannot be found on-line at the SDO's website, a 0 is applied. Of course, information stated in official documentation can differ from the actual, real-life situation. This bias can be bypassed by doing observations during meetings et cetera, but this method comprises easily accessible information. Background information on the workshop results and descriptions of the variables can be found on [16].
<table><caption>**Table 2.** Model derived in this research</caption><thead><tr><th>Criterion</th><th>variables</th><th>Score</th></tr></thead><tbody><tr><td>1. Open meeting</td><td>1 No entrance fee</td><td>0/1/2</td></tr><tr><td rowspan="3">All may participate in the standards development process.</td><td>2 Suitable meeting location</td><td>0/1/2</td></tr><tr><td>3 Open for all</td><td>0/1/2</td></tr><tr><td>4 Open calander</td><td>0/1/2</td></tr><tr><td>2. Consensus</td><td>1 Open and stated decision process</td><td>0/1/2</td></tr><tr><td rowspan="4">During a meeting all interests are discussed and agreement found, no domination.</td><td>2 Procedure if no consensus reached</td><td>0/1/2</td></tr><tr><td>3 Equal voting rights</td><td>0/1/2</td></tr><tr><td>4 External review</td><td>0/1/2</td></tr><tr><td>5 Open agenda</td><td>0/1/2</td></tr><tr><td>3. Due process</td><td>1 Stated appeal process on technology</td><td>0/1/2</td></tr><tr><td rowspan="4">Balloting and appeals during process may be used to find resolution.</td><td>2 Stated appeal process on process</td><td>0/1/2</td></tr><tr><td>3 Independent chairman</td><td>0/1/2</td></tr><tr><td>4 Higher instance for appeals</td><td>0/1/2</td></tr><tr><td>1 Right regime published</td><td>0/1/2</td></tr><tr><td rowspan="2">IPR related to the standard is available to implementers.</td><td>2 Level of IPR (free vs. patents)</td><td>0/1/2</td></tr><tr><td>3 Reciprocal licenses</td><td>0/1/2</td></tr><tr><td>5. One world</td><td>1 Fit with other standards</td><td>0/1/2</td></tr><tr><td>The standard is used for the same capability, worldwide.</td><td>2 Location-independent</td><td>0/1/2</td></tr><tr><td>6. Open documents</td><td>1 Open drafts</td><td>0/1/2</td></tr><tr><td rowspan="5">Standards committee drafts and completed standards documents are easily available for implementation and use.</td><td>2 Open specifications</td><td>0/1/2</td></tr><tr><td>3 Open meeting notes</td><td>0/1/2</td></tr><tr><td>4 Open procedural documents</td><td>0/1/2</td></tr><tr><td>5 Open redistribution</td><td>0/1/2</td></tr><tr><td>1 Back- and forwards compatible</td><td>0/1/2</td></tr><tr><td>7. Open interface</td><td>2 Implementations compatible with specification</td><td>0/1/2</td></tr><tr><td>Standards supports migration and allows proprietary advantage but standardized interfaces are not hidden or controlled.</td><td>1 Conformance testing</td><td>0/1/2</td></tr><tr><td rowspan="4">Standards is equipped with objective conformance mechanisms for implementation testing and user evaluation.</td><td>2 Conformance validation</td><td>0/1/2</td></tr><tr><td>3 Conformance certification</td><td>0/1/2</td></tr><tr><td>4 Disability support</td><td>0/1/2</td></tr><tr><td>1 Support throughout life cycle</td><td>0/1/2</td></tr><tr><td>9. On-going support</td><td>1 Coverage of other dimensions during change</td><td>0/1/2</td></tr><tr><td>Standards are supported until user interest ceases.</td><td></td><td></td></tr><tr><td>10. Open change</td><td></td><td></td></tr><tr><td>All changes in the standards are presented and agreed</td><td></td><td></td></tr></tbody></table>
# 5 Case study

To examine to usability and applicability of the operationalized model, it was tested in a case study. For this purpose the open standard STuF was used. STuF is an abbreviation of standard exchange format and it is an open standard which is used by the Dutch government, especially by municipalities, for the purpose of exchanging basic civilian information (like addresses) within and between information systems [14]. STuF is on the list 'comply-or-explain' standards as discussed in chapter two [12]. More information about STuF can be found in [19].

<table><caption>**Table 3.** Scores of STuF using the model</caption><thead><tr><td>Criterion</td><td>Average score</td><td>Criterion</td><td>Average score</td></tr></thead><tbody><tr><td>Open Meeting</td><td>1</td><td>Open Documents</td><td>1</td></tr><tr><td>Consensus</td><td>1.8</td><td>Open Interface</td><td>0.5</td></tr><tr><td>Due Process</td><td>0.4</td><td>Open Access</td><td>0.5</td></tr><tr><td>Open IPR</td><td>2</td><td>On-going Support</td><td>2</td></tr><tr><td>One World</td><td>1</td><td>Open Change</td><td>1.25</td></tr><tr><td>Total average</td><td>1.1</td><td></td><td></td></tr></tbody></table>

The results are based on information published in official documentation. The official documentation was found on the website of EGEM-ITEAMS, which is the SDO of STuF [14][8][9]. We chose to measure the openness of STuF based only on official on-line documentation since this contributes to the openness of a standards. Maybe the openness of STuF as mentioned in documentation differs from the actual openness in practice. This could be overcome by using other techniques like interviews or observations, but these techniques are very intensive and are therefore not appropriate to use in our framework. Some variables are 0, simply because no explicit information could be found in documentation. In practice it could be that a variable is actually implemented correctly, but since it is not documented it scores a 0. Explicit description in documentation contributes to the openness of a standard. The results show that the STuF standards scores well in Open IPR and On-going Support, these are effectively addressed in the STuF standard. Of concern is the Due Process, which scores a 0.4. This can be explained by the fact that in case of disputes no balloting procedures are described. As the adoption STuF is growing, the likelihood of disputes of any kind is rising. Other points of concern are Open Interface and Open Access, both scoring a 0.5. Issues with version management and compatibility are related to score on Open Interface, while the absence of conformance services are related to Open Access.

# 6 Conclusions

This research shows that standards which – by their SDO – are claimed to be open, are not always fully open. These standards may actually be open on several, but not
all, dimensions. For the Dutch government the model which is constructed in this research is useful, because standards that are claimed to be open can be analyzed for their degree of openness. This avoids situations in which the policy (use of open standards) is not in line with practice (open standards that are in fact not that open). In this case, STuF scores a 1.1, whereas this is one of the most important semantic open standards used within the Dutch government. Usage of the model indicates which aspects of STuF can be made more open, and which aspects are really open. The study provides a useful 'checklist' for government bodies in determining whether to put a standard on the 'comply or explain, and commit' list. This narrows the gap between policy on open standards and actual practice.

This research is also useful for SDO's who want to improve the openness of their standards or to have at least a certain idea which parameters are adaptable to influence the openness of their standards. As said earlier, open standards do not always have to be the open in all dimensions, since certain dimensions are not that important in a specific situation. But, apart from the outcomes, all dimensions related to openness should at the very least be discussed.

Next to having a better view on the openness of a standard the model can be used to compare the openness of standards. For example, if an organization chooses to only use open standards for a specific situation, this model could be used to make a well-based choice for an open standard when competing open standards might exist. But also the scoring of the STUF standard would be more valuable when compared to the scoring of other standards.

Another useful application of this model is to rank SDO's organizations. Often, these organizations are concerned with setting standards for the use in public sector situations. For government organizations these standards are often mandatory by law. It might be interesting how 'closed' these standards actually are. In certain cases the policy on interoperability might collide with legal obligations. If a government wants to stimulate the use of open standards, this model could be useful by making visible how closed certain mandatory standards really are. This might add momentum to open standards and interoperability.

# 7 Further research

This research is a first attempt at operationalizing the dimensions of openness, focusing on the applicability of such an operationalized model. Further improvements and underpinning of the model is needed, including improvement of the measurement scales. For example, more research can be conducted on the degree of mutual exclusiveness and collective exhaustion of the coverage of open standards by the ten dimensions. Also more case studies would be welcome to iteratively improve the model and to gather a database with results.
References

1. Netherlands Open In Connection (NoiV), http://noiv.nl/

2. Bureau Forum Standardization (BFS), http://www.open-standaarden.nl

3. Rosen, L. Open Standards, http://www.rosenlaw.com/DefiningOpenStandards.pdf

4. Andersen, P. 2008. Evaluation of Ten Standard Setting Organizations with Regard to Open Standards. Special study, IDC 2008

5. Krechmer, K. 2008. Open standards: a call for change. IEEE Comm. Mag. 47, 5 (May. 2009), 88-94.

6. Langley, A. 1999. Strategies for theorizing from process data. Academy of Management Review, 24: 691-710.

7. IDABC. Documentation on the European Interoperability Framework, http://ec.europa.eu/idabc/en/document/3473

8. EGEM-iteams: Beheermodel StUF, www.egem-iteams.nl/system/files/StUF+Beheermodel+en+releasebeleid.pdf

9. Bureau Forum Standardization: Expert advice StUF, http://www.egem-iteams.nl/system/files/Expertadvies_StUF_0.pdf

10. Vries, H. de. 1999. Standards for the Nation. Kluwer Academic Publishers, Dordrecht

11. NOiV. An action plan for the use of Open Standards and Open Source Software in the public and semi-public sector,

12. BFS. List of 'comply or explain, and commit' standards, http://www.open-standaarden.nl/fileadmin/os/documenten/OS%20lijst%20open%20standaarden%20voor%20pas%20toe%20of%20leg%20uit.pdf

13. Perens, B. Open Standards: Principles and Practice http://perens.com/OpenStandards/Definition.html

14. EGEM-iteams: website STuF, www.egem-iteams.nl/stuf

15. Fomin, V.V., Pedersen, M.K., and de Vries, H.J. 2008. "Open Standards and Government Policy: Results of a Delphi Survey," Communications of the Association for Information Systems (22:April), pp 459-484.

16. NOiV: Wiki on Open Standards, http://wiki.noiv.nl/xwiki/bin/view/NOiV/Open+Standaarden

17. Rada, R., Berg, J. 1995. Standards: Free or Sold?, Communications of the ACM (38:2), pp 23-27.

18. Zhu, K., Kraemer, K.L, Gurbaxani, V., Xu, S.X. 2006. Migration to Open-Standard Interorganizational Systems: Network Effects, Switching Costs, and Path Dependency. MIS Quarterly (30:Special issue/August).

19. EGEM-ITEAMS. Intelligent standards for municipalities, http://egem-iteams.nl/system/files/Boekje_verdieping_Malmo.pdf

20. Decker, S., Melnik, S., Van Harmelen, F., Fensel, D., Klein, M., Broekstra, J., Erdmann, M., Horrocks, I. (2000). Semantic Web: The roles of XML and RDF. IEEE Internet Computing, 4 (5), pp. 63-74
Appendix – In-depth scoring of STuF

<table><thead><tr><td>Variable</td><td>0/1/2</td><td>Elaboration</td></tr></thead><tbody><tr><td>1.1</td><td>0</td><td>No information found in both [9] and [8]</td></tr><tr><td>1.2</td><td>0</td><td>No information found in both [9] and [8]</td></tr><tr><td>1.3</td><td>2</td><td>In principle open for everyone who's interested, [9] page 10, [8] page 7, [8] page 9</td></tr><tr><td>1.4</td><td>2</td><td>Meeting dates can be found on line.[8] multiple pages in appendix A</td></tr><tr><td>2.1</td><td>2</td><td>Decision process is thoroughly described, [8] page 7 [8] multiple pages in appendix A</td></tr><tr><td>2.2</td><td>2</td><td>Majority decides if no consensus, [8] multiple pages in appendix A</td></tr><tr><td>2.3</td><td>1</td><td>Every stakeholder has voting rights, but equality is not explicitly stated, [8] page 16</td></tr><tr><td>2.4</td><td>2</td><td>External reviewers are invited to comment, [8] page 8, [8] page 15</td></tr><tr><td>2.5</td><td>2</td><td>Attendees can put topics on agenda, [8] page 11</td></tr><tr><td>3.1</td><td>2</td><td>Appeals are possible in case of technology issues, [8] page 12 [8] page 16</td></tr><tr><td>3.2</td><td>0</td><td>No information found in both [9] and [8]</td></tr><tr><td>3.3</td><td>0</td><td>No information found in both [9] and [8]</td></tr><tr><td>3.4</td><td>0</td><td>No information found in both [9] and [8]</td></tr><tr><td>3.5</td><td>0</td><td>No information found in both [9] and [8]</td></tr><tr><td>4.1</td><td>2</td><td>Rights regime is published, [8] page 9</td></tr><tr><td>4.2</td><td>2</td><td>Royalty-free, [8] page 9</td></tr><tr><td>4.3</td><td>2</td><td>Licenses on (subsets of) the standard are reciprocal, [8] page 9</td></tr><tr><td>5.1</td><td>2</td><td>The fit with related standard is described, [9] page 11 [8] page 8</td></tr><tr><td>5.2</td><td>0</td><td>No information found in both [9] and [8]</td></tr><tr><td>6.1</td><td>1</td><td>Drafts are published, but it's unsure if this is publicly [8] page 8</td></tr><tr><td>6.2</td><td>2</td><td>All specification can be found online, [8] page 9, [8] page 8</td></tr><tr><td>6.3</td><td>0</td><td>No information found in both [9] and [8]</td></tr><tr><td>6.4</td><td>2</td><td>Procedural documents can be found online, [8] page 9, [8] multiple pages in appendix A</td></tr><tr><td>6.5</td><td>1</td><td>Reciprocal licenses (4.3) imply this, but not explicitly stated</td></tr><tr><td>7.1</td><td>0</td><td>No information found in both [9] and [8]</td></tr><tr><td>7.2</td><td>2</td><td>A conformance certificate is applied. [8] page 17</td></tr><tr><td>8.1</td><td>0</td><td>No information found in both [9] and [8]</td></tr><tr><td>8.2</td><td>0</td><td>No information found in both [9] and [8]</td></tr><tr><td>8.3</td><td>2</td><td>Conformance certification is available, page 11 [8]</td></tr><tr><td>8.4</td><td>0</td><td>Not found</td></tr><tr><td>9.1</td><td>2</td><td>Standard is supported throughout standards life cycle [8] page 19</td></tr><tr><td>10.1</td><td>0</td><td>For example not the dimensions that score low</td></tr></tbody></table>