# A Framework for Interoperability Testing in Pan-European Public Service Provision

Ansgar Mondorf, Maria A. Wimmer, Daniel Reiser

► To cite this version:

Ansgar Mondorf, Maria A. Wimmer, Daniel Reiser. A Framework for Interoperability Testing in Pan-European Public Service Provision. 12th International Conference on Electronic Government (EGOV), Sep 2013, Koblenz, Germany. pp.188-199, 10.1007/978-3-642-40358-3_16. hal-01490906

HAL Id: hal-01490906

https://inria.hal.science/hal-01490906v1

Submitted on 16 Mar 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# A Framework for Interoperability Testing in Pan-European Public Service Provision

Ansgar Mondorf, Maria A. Wimmer, Daniel Reiser

Institute for Information Systems Research,
University of Koblenz-Landau, Universitätsstr. 1, 56070 Koblenz, Germany
{mondorf | wimmer}@uni-koblenz.de

**Abstract.** A major challenge of interoperability projects and initiatives is to validate that different implementations work together and are compliant with underlying standard specifications. Interoperability testing can ensure that required end-to-end functionality is adequately fulfilled and all systems are implemented in conformity with existing standards. Adopters of standards have different methods to prove fulfillment of interoperability requirements. This often results in different efficiency, different quality and a lack of conformity. In this paper, we present the interoperability testing framework used in PEPPOL (Pan-European Public Procurement Online). The framework has supported the project well in establishing various production pilots that interoperate together in the field of e-procurement. It thereby aggregates different testing perspectives ranging from conformance testing to compliance testing and provides guidelines how to prove implementations through testing scenarios. Consequently, the key research question in this paper is how interoperability initiatives can prove that different implementations are compliant with underlying standard specifications and how interoperability can be ensured among different implementations beyond the technical approval mechanisms provided through conformance testing.

**Keywords:** Pan-European Public Service Provision, Interoperability Testing, Conformance Testing, Compliance Testing, Public E-Procurement

## 1 Introduction

Interoperability projects and initiatives often face the problem that sector specific scenarios are widely adopted by users, but the establishment of interoperability between various communities and implementations still remains cumbersome. Several authors have emphasized that greater interoperability within and across private and public sector entities could be achieved when more advanced testing methodologies and practices are used. Common guidelines for interoperability testing could foster standards adoption and could help to achieve better conformance and compliance with the underlying standards, specifications and agreements. They increase overall coherence, consistency and quality of standards and provide active support to their imple-
mentation. At the same time the risk of fragmentation, duplication and conflicting
testing efforts could be reduced [1,2].

Most authors agree that interoperability can be described upon basic characteristics which can be generally described as the ability of different ICT systems to communicate and interact with each other as well as to exchange information [5,6]. Conformance testing can be used to prove the ability of ICT system to communicate and interact together as it describes the process for verifying that an implementation of a specification fulfills its requirements both in terms of choreography of collaborations and in terms of information constraints on information elements<fnref n="1" />. Nevertheless there are broader definitions of interoperability which also refer to the ability of organizations to co-operate seamlessly on the basis of common processes, business rules and agreements [8]. According to the European Interoperability Framework (EIF) interoperable specifications shall address legal, organizational, semantic and technical aspects of a business process. Although the development of European public services mostly includes the exchange of data between ICT systems, interoperability can be understood as a wider concept, encompassing for example the ability of private and public sector entities to work together towards mutually agreed goals which are beneficial to all parties involved [3]. Taking this wider interoperability scope into account, it seems obvious that conformance testing cannot be the only means to test interoperability. In fact conformance testing can be seen as a prerequisite for interoperability testing. Interoperability testing is executed by operating implementations enforcing them to interoperate following a specific behavior [1] and thereby addressing testing activities on legal, organizational, semantic and technical level.

Interoperability testing provides a wider scope than conformance testing as it also describes methods, processes and tools that are required by adopters to claim compliance. This not only includes the technical process of testing the conformance of document instances but also the proof that agreements are singed or legal perspectives of a system have been considered correctly during the implementation process. Interoperability testing therefore has to define a wider set of activities to ensure that systems work properly together. This paper subsequently investigates theories and related work on conformance and compliance testing in section 2. Thereby, the differences between conformance and compliance testing as well as how both fulfill a distinct need in the interoperability testing are argued. The interoperability testing framework used to support the implementation of PEPPOL building blocks through conformance and compliance testing methods is presented in section 3. The methods were aggregated to a framework to prove components against the PEPPOL specifications and agreements. While conformance testing is a well-known method in standards implementation and technical approval it does not cover all aspects required to ensure interoperability on an organizational, semantic and legal level in complex environments. The paper therefore concentrates on the use of compliance testing methods. Section 4 exemplifies the use of the framework and its testing methods in the context of the Virtual Company Dossier (VCD), a PEPPOL building block for electronic ten-

<footnote n="1">¹ CEN Workshop Agreement (CWA 16073-1): Report on Requirements: Conformance and interoperability testing. European Committee for Standardization, Brussels (2010)</footnote>
dering. A number of guidelines, scenarios and underlying specifications clarify how
the framework was practically implemented in the project. The conclusions in section
5 assess the benefits of a coordinated interoperability testing approach and make sug-
gestions for further research directions.

## 2 Theories and related work

The major elements relevant to interoperability testing are conformance testing and
compliance testing. In conformance testing, a specification comprises all agreements
done by a community in particular the underlying standards or profiles, which repre-
sent subsets or interpretations of that specification. From an architectural perspective,
agreed specifications can be described as follows:

*Profiles* describe specific parts of a process, where bilateral (or multilateral) communications are executed. Together they form the interfaces of that process apart from internal or back-office logic. A *process* consists of collaborations, which define the choreography of transactions between parties. These parties may exchange documents (e.g. a XML message) among each other. Each *document instance* refers to an underlying transaction data model defined in the profile. *Transaction data models* consist of several *information elements*, which may be further restricted through information constraints. An implementation of such a data model may require to be tested against the data model to proof its conformance. Conformance testing describes the process for verifying that an implementation of a specification fulfills the requirements of the profile, both in terms of choreography of collaborations and in terms of information constraints on information elements¹.

According to Gebase et al. [4] conformance testing is a black-box test, where the tester has no knowledge about the structure and code of the implementation. The testing environment, which aggregates the test capabilities, is named test bed. The test bed consists of a set of test assertions, which formally define the requirements or information constraints to be tested. Test assertions are intended to define constraints such as the existence or value of information elements. They may also describe cases, where a value depends on another value, a function or a code list¹. Validators and validation artifacts such as the underlying XML-Schema, Code Lists or Schematron files are summarized within the test bed and they identify, whether a given document instance or test item complies with the defined requirements. Conformance testing may be executed through valid and invalid test items, which are sent to the test bed and thereby tested against the test assertions using the validation components of the test bed. The result is the divergence of the test item to the underlying specification or standard. Invalid test items are often used to test the validation component itself [1,4].

In the context of interoperability testing, conformance testing can be seen as a pre-
requisite for interoperability testing. Interoperability testing shall ensure that different
systems work together on all levels. Interoperability testing is executed by operating
implementations enforcing them to interoperate following a specific behavior [4].
Ivezic et al. describe interoperability testing as the process for verifying that several
implementations can interoperate together while conforming to one or more specifica-
tions [1]. The authors remark that conformance testing does not guarantee interopera-
bility and that in turn interoperability testing does not substitute the conformance testing. In fact, interoperability testing is usually more cost-intensive than conformance testing, since time and human efforts have to be spent in order to coordinate and set up an appropriate environment for interoperability testing. However, interoperability testing is usually more successful and less costly when the conformance of implementations has been tested first. Thus interoperability testing can largely benefit from conformance testing [1].

The term compliance testing summarizes methods beyond conformance testing, focusing on legal and organizational aspects. All relevant methods are aggregated within the interoperability testing framework, which was conceptualized in the PEPPOL project. While the aspects of conformance were rather clear during the conceptualization, compliance testing had an unclear scope with less approaches and guidelines available. Conformance testing deals with the technical and syntactic interoperability on established standards such as XML, while covering certain semantic aspects at the same time (e.g. multilingual code list values). Thus for semantic interoperability there are some concepts and methods available, but for legal and organizational interoperability, it is by far less obvious how a standardized concept and method could look like for compliance testing, and who could develop and establish appropriate standards [7]. The paper at hand contributes to clarify and structure the full scope of interoperability testing beyond the more technically-oriented conformance testing. The next section presents the framework for interoperability testing as developed in PEPPOL.

## 3 A Framework for Interoperability Testing

The framework for interoperability testing provides a structured approach of testing software distributions. It can be used to verify that implementations comply with specifications and requirements. It also ensures that systems considering legal and organizational cross-border aspects are appropriate. Figure 1 depicts the overall framework for interoperability testing, distinguishing main test groups and specific test types that define the outline of test executions.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Test groups and test types of the framework for interoperability testing</figcaption></figure>

Conformance Testing ensures that software components are able to generate and understand correct document instances. Compliance testing is particularly used to verify that legal, organizational and semantic aspects are correctly implemented. It thereby focusses on common denominators across all implementations such as the use and
testing of central components as well as the execution of end-to-end tests between implementations and pilots. While compliance testing focuses on a domain specific range of issues relevant to a process (e.g. legal and organizational peculiarities, agreements, central service provision), conformance testing strictly focuses on the adherence to a standard. Both test groups are detailed subsequently.

## 3.1 Conformance Testing

The aim of conformance testing is to test whether software components are able to generate and understand correct document instances. Document instances in the PEPPOL VCD context are correct if they are valid along the aspects listed in table 1.

<table><caption>**Table 1.** Validation aspects of VCD conformance testing</caption><thead><tr><th>Aspect</th><th>Description</th></tr></thead><tbody><tr><td>Document structure</td><td>Correctness of the structure and data elements of an XML instance as defined in the corresponding XML schema</td></tr><tr><td>Element cardinalities</td><td>Optional elements MAY exist, mandatory elements MUST exist. Prohibited elements will not be part of a profile XML schema.<br/>— 0..1 = optional, zero or one occurrence<br/>— 0..* = optional, zero or more occurrences<br/>— 1 = mandatory, one occurrence<br/>— 1..* = mandatory, one or more occurrences</td></tr><tr><td>Data types</td><td>String, integer, float, date, time, Boolean</td></tr><tr><td>Value ranges</td><td>— Length (for elements of type string)<br/>— Range (for elements of types integer, float, date and time)</td></tr><tr><td>CodeLists</td><td>Correct use of values as defined in CodeLists</td></tr><tr><td>References</td><td>— Correct file references in a document<br/>— Existence of VCD sub folders for economic operators defined in the VCD Package meta-data file.<br/>— Correct element references.</td></tr><tr><td>Value patterns</td><td>Correct values and format of specific element values, e.g. UUID: random UUID; Date: yyyy-mm-dd; Time: hh:mm:ss</td></tr></tbody></table>

Table 2 describes the different test types of conformance testing, which cover the validation aspects listed above.

<table><caption>**Table 2.** Conformance test types</caption><thead><tr><th>Test type</th><th>Description</th><th>Rationale</th><th>Validation artefacts</th></tr></thead><tbody><tr><td>XML schema validation</td><td>XML schema validation of document instances against</td><td>As a minimum requirement of validation, document instances have</td><td>XML schema files: Define the technical structure of a document instance, including allowed elements, cardinalities,</td></tr></tbody></table>
<table><thead><tr><th></th><th>the specified XML schemas.</th><th>to conform to an XML schema.</th><th>data types and value ranges.</th></tr></thead><tbody><tr><th scope="row">Schematron validation</th><td>Schematron validation of document instances against Schematron rules.</td><td>Used for additional validation against rules not covered by the basic schema validation.</td><td>Schematron files: Define Schematron validation rules</td></tr><tr><th scope="row">Information constraints validation</th><td>Validation of document instances against additional information constraints derived from business rules.</td><td>It is necessary to test such constraints that cannot be covered by XML schema and Schematron validation, e.g. file references within an XML instance.</td><td>Context value associations (CVA) files: Express the relationship between controlled vocabularies (e.g. CodeLists) and document elements.<br/>Test System rule definition: Define validation rules for a Test System.</td></tr></tbody></table>

Figure 2 illustrates the phases of the conformance testing methodology. These steps allow a structured and organized testing of document instances.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Conformance testing phases</figcaption></figure>

The first phase covers the definition of validation requirements, business rules and information constraints that define the rules and constraints against which document instances are being validated. In order to prepare the application profiling phase, all relevant requirements and rules have to be formally defined through a schema profiling tool (e.g. SchemaProf<fnref n="2" />). Application profiles provide the technical basis for implementing the tests. The step of application profiling shall lead to a concrete set of test system configurations that can be processed by validation software. For the actual testing of document instances, each application profile has to be implemented as a test system. A Test System is a tool, which allows configuring test systems. It provides an executable file, through which the validation process can be started through a document instance of the test object. A detailed report is generated by the Test System after the test execution [2]. The results of test runs have to be documented in a common way. Hence, test protocols need to be created for the different test executions, and these protocols have to be stored in a central place. Test case management software (e.g. pDAF<fnref n="3" />) supports users to define common test cases and test tracks as well as to report and document executions and results.

<footnote n="2">² http://iwm.uni-koblenz.de/schemaprof/</footnote>

<footnote n="3">³ http://peppol.phloc.com/</footnote>
## 3.2 Compliance Testing

In practice, the terms compliance and conformance are often conflated. Compliance testing means checking the behavior of a system at runtime to determine, if it behaves as desired. In contrast, conformance testing describes the ability to operate in the way defined by a specification or more precisely to test whether software components are able to generate and understand correct document instances. To make specifications suitable for implementation and interoperation, they often require to be amended through further agreements and interpretations, thus reflecting specific issues of countries and/or communities. In public procurement many processes are complex due to their legal nature. Standard specifications may reduce/exclude certain complexity aspects by assuming that an adopter/community will interpret the specifications (e.g. by defining code list and IDs) and manage varying practices in legislations (e.g. binding legal use of certified copies vs. scanned copies) correctly. In consequence, conformance testing has to focus on common specifications, their interpretation and agreements beyond the document and process level targeted by conformance testing. The framework for interoperability testing introduced in figure 1 therefore defines several compliance test types. Table 3 explains these elements by describing test types, the description and rationale of the test type and respective validation artifacts.

<table><caption>**Table 3.** Compliance test types</caption><thead><tr><td>Test type</td><td>Description and Rational</td><td>Validation artefacts</td></tr></thead><tbody><tr><td>Legal organisational and semantic compliance testing</td><td>Describes whether components fulfil the agreed specified legal, organisational and semantic behaviour. Obligatory legal, organisational and semantic rules and logic need to be checked in order to determine whether systems adequately comply with underlying legislation and organizational practices.</td><td>Document instances: Document instances are used to validate whether they comply with underlying legal, organizational and semantic requirements.<br/>Test Cases: Appropriate test cases define representative or exceptional behaviour in certain legal, semantic and organisational aspects (depending on what to test).</td></tr><tr><td>Proof of Concept, Test and Production pilots<sup>4</sup></td><td>Pilots with increasing complexity are used to specifically address legal, organisational, semantic and technical issues in a narrow and artificial setup. Proof of concept and test pilots are used to determine how the system behaves with real data but artificial set-up.</td><td>Pilots: Pilots are selected and complexity is raised gradually in order to convey the pilot from Proof of concept pilot (narrow set up) and test pilot (real data/artificial process) into productive pilot environment (real data/real process).</td></tr><tr><td>End-to-end scenario testing</td><td>End-to-end scenario testing emphasises on the integration among systems, applications and pilots to test system dependencies. The focus of end-to-end testing scenarios lies on interfaces and dependencies between pilots.</td><td>Scenarios: End-to end scenarios ensure to test the fully integrated pilots together and to check that they act as a whole. Different pilots may be combined to build a scenario for end-to-end testing.</td></tr></tbody></table>

<footnote n="4">⁴ CEN Workshop Agreement (CWA 16073-4): Part 4: Evaluation guidelines for testing and piloting. European Committee for Standardization, Brussels (2010)</footnote>
One could argue that the distinction between system testing and compliance testing is rather unclear since both of them focus on functional and non-functional aspects of a system. The distinction in PEPPOL was done on the basis of system dependencies and what systems shall have in common according to the PEPPOL specifications and agreements. This may include obligatory interfaces or a commonly agreed legal, organizational and semantic behavior of a system. Two further issues shall be denoted at this point: (1) Legal and organizational requirements underlie frequent changes, which result from modifications of national or European legislations. Legislative modifications therefore have to be closely monitored, and changes have to be implemented and tested in accordance with these modification. (2) Central components that are interfaced to many pilots or implementations also take a specific role. Responsibilities and liabilities for testing these central components have to be defined, since these components are typically used by all or many pilot implementations.

In summary major elements of compliance testing are the adequate fulfillment of legal, organizational and semantic requirements, the smooth conversion of implementations into productive environments and the testing of pilots, systems and components in cooperation raising the complexity of the business process gradually.

# 4 Exemplifying Conformance Testing with PEPPOL VCD

The PEPPOL project addresses the development and implementation of technology standards to align business processes for electronic public procurement across Europe. PEPPOL's vision is to enable businesses to communicate electronically with any governmental institution in Europe for executing public procurements. PEPPOL has developed standards-based components and tools to support interoperable e-procurement in different stages of the value chain. The Virtual Company Dossier (VCD) is a component in the pre-awards stage (see e.g. [9]) and herewith serves as an example to demonstrate the underlying models of the ICT architecture and the approach to interoperability testing. PEPPOL has decided to adopt a generic Enterprise Interoperability Architecture (EIA) to organize the artifacts in a consistent and flexible way. In this paper, we rely on the structure proposed by the PEPPOL EIA<fnref n="5" /> to reference relevant artifacts.

The Virtual Company Dossier (VCD) has been developed to address the demand for better interoperability in electronic tendering offering a standardized document container to suppliers, which can be used to submit evidences (e.g. attestations and certifications) as part of the qualification process in public procurements [9]. Several VCD implementations had to be tested against the underlying standard specifications. With the PEPPOL VCD, we subsequently demonstrate the approach to interoperability testing in the project. All procurement solutions relevant to PEPPOL have been built on the basis of standards and tools recommended by the CEN/ISSS workshop on Business Interoperability Interfaces (CEN ISSS/WS BII). The CEN/ISSS workshop has published a set of BII (Business Interoperability Interface) profiles, each of them

<footnote n="5">⁵ Cicirello, C., Hayworth, M.: PEPPOL Starter Kit. Austrian Federal Ministry of Finance, Vienna (2011)</footnote>
addressing a unique part of the public procurement process. A BII profile describes the choreography of the business process with a detailed description of the collaborations including roles and responsibilities. Each BII profile is referencing its business transactions to data models, which describe the core information entities on a semantic level. On a syntax level, these data models are bound to XML documents compliant with schemas from international standards such as UN/CEFACT XML and OASIS Universal Business Language 2.0. Thus a BII profile can be seen as a major reference for conformance testing as it aggregates the key requirements, agreements and standards of the underlying business process<fnref n="6" />,<fnref n="7" /> [6,8].

The principal specifications used in PEPPOL are known as PEPPOL Business Interoperability Specifications (PEPPOL BIS). PEPPOL BIS are based on the aforementioned BII profiles and they address additional legal, organizational, semantic and technical aspects of a business process in accordance with the European Interoperability Framework (EIF). A PEPPOL Business Interoperability Specification (BIS) summarizes all amendments which are done to a CEN BII profile. A PEPPOL BIS is more specific than a BII profile because it adds usage-specific agreements of the PEPPOL community. It also provides supporting guidelines how to implement the requirements. A PEPPOL BIS therewith details certain aspects such as the legal scope of a specification. It also adds elements such as validation rules, identifiers and code lists that have been commonly defined in PEPPOL and addresses for example dependencies with central service components. Adopters that want to comply with a PEPPOL BIS have to comply with each interoperability layer. Thus conformance with PEPPOL BIS specifications is only one aspect of interoperability testing which focuses the syntactical and technical level and the capability of applications to generate valid document instances.

So far this paper has described the range and structure of interoperability tests executed in the PEPPOL project. In the following, an overview about the set of supporting artifacts, which are provided to adopters for the establishment and testing of PEPPOL compliant interfaces and products, is given. The artifacts listed below describe the relevant outputs according to the framework for interoperability testing introduced in section 3. They can be accessed via the PEPPOL EIA<fnref n="8" />. As discussed before, the VCD Business Interoperability Specification (BIS) summarizes all major aspects of interoperability testing considering all levels introduced by the EIF. The VCD BIS therewith refers to the following sub-artifacts, which are related to the test types defined for conformance testing in the testing framework:

* Specification and hierarchical data model of the data format of VCD and VCD Package in the form of testable XML schemas for the VCD, VCD Package and the common VCD library (based on UBL 2.0) (available in the PEPPOL EIA under ICT Architecture -> Models and Service Components).

<footnote n="6">⁶ CEN Workshop Agreement (CWA 16073-1): Part 1: Profile overview. European Committee for Standardization, Brussels (2010)</footnote>

<footnote n="7">⁷ CEN Workshop Agreement (CWA 16558 – Annex A): Annex A - Profile Architecture. European Committee for Standardization, Brussels (2012)</footnote>

<footnote n="8">⁸ http://www.peppol.eu/peppol_components/peppol-eia/eia</footnote>
* Specification of codes and values (incl. translations) of all VCD-related classified lists (i.e. code lists) provided as genericode files (available in the PEPPOL EIA under ICT Architecture -> Models and Service Components).

* Abstract business rules and UBL bindings for the VCD and VCD Package and the definition of rules against which VCD and VCD Package instances must be valid (available in the PEPPOL EIA under Conformance and Test -> Models).

The above mentioned artifacts are aggregated into application profiles for VCDs and VCD Packages (available in the PEPPOL EIA under Conformance and test -> Models). SchemaProf, a tool for XML based application profiling has been used to aggregate and describe all modifications described above. The application profiles are used to create correlating VCD Conformance Test System which can automatically detect variations from the aforementioned artifacts. The Conformance Test Systems are generated from application profiles and allow automatic validation of VCD Containers against the rules defined in the VCD data model as well as requirements regarding codes and values and business rules (available in the PEPPOL EIA under Conformance and test -> Service Components).

Apart from the conformance testing artifacts, a number of compliance testing artifacts have been created:

* The ontology governance process is a guideline for setting duties, roles and responsibilities for governing the legal domain ontologies and keeping the central component, the European VCD system, up to date. As all VCD implementations depend upon this central service provision component it is essential that all adopters can rely on the information source. To ensure the quality of the information source a set of ontology quality assurance tools have been developed to establish a well-functioning quality assurance process (available in the PEPPOL EIA under Conformance and test -> Models and Service Components).

* To proof representative or exceptional behavior in certain aspects a set of test case definitions have been defined. Test cases can be aggregated to test-tracks thus complexity can be raised gradually. Test cases can be used to address certain legal, organizational, semantic aspects during process executions but they can also be used for a proof of concept and pilot testing (available in the PEPPOL EIA under Conformance and test -> Models).

* Historical Call for Tenders are used to provide data to the test executions in order to build test pilots that rely on real data within an artificial process set-up (available in the PEPPOL EIA under Conformance and test -> Models).

* A guideline for Contracting Authorities describes how to pilot the VCD in real environments including sample Call for Tender and sample VCD response. Since the VCD solution depends on other tendering processes beyond PEPPOL the full scope of scenarios is analyzed and suitable conditions to build effective VCD scenarios are identified (available in the PEPPOL EIA under Marketing -> Models).
## 5 Discussion and concluding remarks

This paper introduced a generic framework for interoperability testing and exemplified its application through the PEPPOL VCD solution. Interoperability testing helps to overcome a major challenge of interoperability, i.e. it helps to validate that different implementations work together and are compliant with the underlying specifications and standards. Greater interoperability can be achieved through application of interoperability testing methods and practices to foster standards adoption. Common methods and practices increase the coherence, consistency and quality of standards, provide active support to their implementation and reduce the risk of fragmentation, duplication and conflicting testing efforts. In this respect, the paper has shown the different aspects of interoperability testing, separating clearly the concern, approach and goals of conformance testing and those of compliance testing.

The use of conformance testing and compliance testing methods in PEPPOL helped to ensure that VCD solutions co-operate seamlessly on the basis of common processes, business rules and agreements of the project. Whereas conformance testing is a well-structured process that can potentially be executed automatically on any document instance using specifically designed test systems that aggregate all test assertions, compliance testing is a less structured challenge and requires more involvement of resources for designing and executing appropriate tests for the implementation under test. Overall, conformance testing was very helpful in PEPPOL for the technical approval of the system capabilities to generate correct outputs, whereas the compliance testing covered the aspects that arise from the complexity and the intention to use the solution in cross-border scenarios. For simple transactions or services with little legal and organizational constraints involved, conformance testing might be a sufficient method for testing interoperability. Transactions in complex environments with many legal risks, organizational dependencies and multilingual requirements require stronger control and specifically designed test techniques to assure proper implementation and reliability. Within the VCD context the goals of compliance testing were often conflated with other goals such as the maintenance of central service components, which provided decision support on the basis of national and European legislation. Within and beyond the project, it has to be ensured that central components are operated, monitored and updated in a reliable manner and compliance testing can help to establish routines for system changes. Another interesting aspect that was well conducted with compliance testing outcomes was the definition of appropriate and suitable conditions for production pilots. As the VCD solutions only support a specific part of e-tendering, the solutions depend and rely upon other e-tendering processes. In order to establish and use VCD implementations in production, pilot guidelines and scenarios where necessary that clarify the dependencies and interrelations with other transactions, carried out with other, conventional means such as e-tendering platforms, out of the PEPPOL scope.

The paper references accessible artifacts that have been created in PEPPOL to clarify the application of the framework. The artifacts support developers and testers in the process to check and prove VCD implementations against the VCD specifications and underlying standards. In case of new releases, existing implementations can be
easily updated and adjusted using the guidance provided by these artifacts. In this
respect the artifacts also support release management, maintenance and long-term
sustainability. Furthermore adopters have a clear process to claim conformance to
other network participants. These benefits show that a sophisticated framework for
interoperability testing can strongly assist the process to develop and establish pan-
European e-government services.

Future research is needed to elaborate and approve the aspects of interoperability
testing presented in this paper with a particular focus on compliance testing. While
conformance testing is a more mature technical method, which can be adopted easily
by other projects, compliance testing is unclear in its appearance and definition. The
division into legal, organizational and semantic compliance testing, pilots and end-to-
end testing scenarios is a first step to frame the overall scope of compliance testing.
As the results of this paper have been developed in purpose of the PEPPOL solutions,
their universality is not fully given at the moment. Further evidence is needed regard-
ing suitability in other contexts and projects.

References

1. CEN Workshop Agreement (CWA 16408): Testing Framework for Global eBusiness Interoperability Test Beds (GITB). European Committee for Standardization, Brussels (2012).

2. Dahn, I. and Zimmermann, S.: Application Profiles and Tailor-Made Conformance Test Systems. International Journal of IT Standards and Standardization Research (IJITSR), 8 (2), 2010, pp. 60-73, (2010).

3. European Commission: European Interoperability Framework (EIF) for European public services - Annex II - Towards interoperability for European public services, Brussels, (2010).

4. Gebase, L., Snelick, R. and Skall, M.: Conformance Testing and Interoperability: A Case Study in Healthcare Data Exchange. In: Arabnia, H.R., and Reza, H. (Eds.): Proceedings of the 2008 International Conference on Software Engineering Research & Practice, SERP 2008, Las Vegas Nevada, USA, 2 Volumes, pp. 143-151. CSREA Press, 2008.

5. Gottschalk, P. and Solli-Sæther, H.: E-Government Interoperability and Information Re-
source Integration: Frameworks for Aligned Development. Information Science Reference,
Hershey, New York (2009)

6. Kosanke, K.: ISO Standards for Interoperability: A Comparison. In Konstantas, D., Bourrières, J.-P., Léonard, M., and Boudjlida, N., editors, Interoperability of Enterprise Software and Applications, pages 55–64, London. Springer-Verlag (2006).

7. Kubicek, H., & Cimander, R.: Three dimensions of organizational interoperability. In- sights from recent studies for improving interoperability frameworks. European Journal of ePractice (6), pp. 13-14, (2009)

8. Mertins, K., Ruggaber, R., Popplewell, K., and Xu, X.: Enterprise Interoperability
III. Springer-Verlag (2008).

9. Mondorf, A., Wimmer, M.A.: The European VCD System: Facilitating Public Procurement Through Criteria to Evidence Mapping. In: Berleur, J.; Hercheui, M.; Hilty, L. (Eds.): What kind of Information Society? Governance, Virtuality, Surveillance, Sustainability, Resilience. Proceedings of 9th IFIP TC 9 International Conference, HCC9 2010, Springer Verlag: Berlin. pp. 73 - 85.