# Evaluating Second Generation Cross-Country Open Legal Data Infrastructures Using Value Models

Charalampos Alexopoulos, Euripidis Loukis, Shefali Virkar

► To cite this version:

Charalampos Alexopoulos, Euripidis Loukis, Shefali Virkar. Evaluating Second Generation Cross-Country Open Legal Data Infrastructures Using Value Models. 13th International Conference on Electronic Participation (ePart), Sep 2021, Granada, Spain. pp.180-197, 10.1007/978-3-030-82824-0_14. hal-04014067

HAL Id: hal-04014067

https://inria.hal.science/hal-04014067

Submitted on 3 Mar 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Evaluating Second Generation Cross-Country Open
Legal Data Infrastructures Using Value Models

Charalampos Alexopoulos<sup>1[0000-0002-6610-0675]</sup>, Euripidis Loukis<sup>1</sup> and Shefali Virkar<sup>2[0000-0003-0701-7684]</sup>

<sup>1</sup> University of the Aegean, Greece
alexop@aegean.gr; eloukis@aegean.gr

<sup>2</sup> Department for E-Governance and Administration, Danube University Krems, Austria
shefali.virkar@donau-uni.ac.at

**Abstract.** Access to legal information is of critical importance to socio-economic and political activity. Hence, the provision of capabilities to search for, locate and retrieve legal information in an efficient and structured manner to governments, businesses, lawyers and citizens is highly valuable. In order to satisfy the requirement for legal information, a ‘first generation’ of open legal data infrastructures that focus on providing access to national legislation and offer some basic functionalities for the providers and the users of these data, is already available in many countries. However, rapid globalization, the emergence of supra-national unions of nations, the advent of advanced data processing capabilities, and the ever-increasing complexity of legislation as it comes to mirror modern life have all contributed to the development of more advanced ‘second generation’ open legal data infrastructures that facilitate access to legal information from multiple national legal frameworks in multiple languages. In the face of the advent of these new technologies, it becomes vital to understand how their performance could be evaluated. This paper presents and validates a methodology for evaluating the emerging second generation of Big Open Linked Legal Data (BOLLD) e-infrastructures based on the concept of ‘value model’ estimations from users’ evaluation ratings. The proposed approach advances beyond the traditional Information Systems (IS) evaluation approaches, as it includes assessments not only of the magnitudes of a wide range of types of value generated by such an infrastructure, but also of the relations among them. The proposed model, therefore, enables a deeper understanding of the whole value generation mechanism, and also can provide a rational definition of priorities for system improvement based on the capabilities offered to users. A first application of the developed approach is made for the evaluation of an advanced second generation BOLLD e-infrastructure developed as part of the European project ManyLaws, leading to both interesting insights as well as improvement priorities.

**Keywords:** open data; legal data; legal informatics; e-infrastructures; evaluation; value model.
# 1 Introduction

Access to legislation is of critical importance for all economic and political activity. Therefore, the provision of capabilities to search for, find and retrieve required legal information in an efficient and structured way to governments, businesses, lawyers and citizens is highly valuable. In order to satisfy differing needs for legal information many countries have developed a ‘first generation’ of open legal data infrastructures that focus on facilitating access to national legal legislation, and which offer a number of basic functionalities for the providers and the users of these data. However, the internationalization of economic activity, along with the emergence of supra-national unions of countries has created a demand for access to legislation not only of the home country, but also of many other nation states, and to international legislation. Simultaneously, advanced processing and analysis of this massive information – big data – have been developed in order to make it practically manageable and highly useful, and to enable such novel services as comparative analyses among countries in order to identify similarities and differences among them. Furthermore, legislation, both national and international, has become more complex, extensive, and dynamic, responding to the increasing complexity and dynamism of economic and social life as well as its problems and challenges (e.g., economic, social, political and environmental crises, digital transformations in both the private and the public sector of the economy and resulting disruptions, etc.). These factors have necessitated the development of a ‘second generation’ of more advanced open legal data infrastructures which enable access to legal information from many different countries, are oriented towards the elimination of the language barrier, and also offer more advanced processing capabilities and permit the analysis of this massive and dynamic legal information.

One of the most important supra-national unions of countries is definitely the European Union (EU). The main vision of the EU is to establish a well-functioning Digital Single Market, wherein European citizens can move freely and trade with their counterparts in other EU member states [1]. Digital transformation, and in particular the development, deployment and uptake of disruptive technologies, lies at the heart of the European approach to empower citizens and facilitate seamless business transactions [1]; it has a strong potential to foster economic development, but at the same time poses serious challenges and made necessary the need for effective regulations based on sound legal frameworks. The centrality of the role played by legal information in decision-making within different political, social, and economic settings entails unhindered access to the legal framework of the EU, as well as its member countries, which is vital for attaining this vision of unfettered cross-border mobility and trade. In other words, a significant prerequisite of a well-functioning Digital Single Market, within which Europeans can live, work and exploit new business opportunities, is a comprehensive knowledge of the legal and policy framework that circumscribes their actions, both in their countries and in other member states as well [2].

The European legal system is multi-layered and complex, as well as extensive, with large quantities of legal documentation having been produced since its formation. However, although European society is overwhelmed by an overload of legal information, only legal experts possess the capacity and wherewithal to follow and comprehend the
latest legislation and policy evolutions and outcomes produced by ministries, parlia-
ments, and courts at different levels of government (local, national and supra-national)
– and even they oftentimes find it difficult to locate meaningful, relevant legal data [2].
A large amount of information about laws that apply in the EU member countries cur-
rently remains fragmented across multiple national databases, hidden within inaccessi-
ble systems, and scattered across public data silos. Mass customization tools, such as
advanced legal information retrieval systems, offer some degree of solution to these
problems, as they can help to sort, filter and present legal information in a logical and
user-friendly manner. These tools and associated service have the potential to make
legal information more easily accessible and comprehensible to businesses and lay us-
ers, reducing the need for recourse to expensive legal expertise [3].

In this direction the advances in the area of legal information retrieval, part of the burgeoning field of legal informatics, which can be defined as the science of information retrieval applied to legal texts - including legislation, case law, expert commentary, and scholarly works, are quite useful [4]. Accurate legal information retrieval is important in order to facilitate access to current legal documents by different groups of actors in the economy and the society. At a practical level, the retrieval of appropriate legal information, and its adequate comprehension, is at once a must and a challenge for European citizens, businesses, local administrations, national governments and institutions. In order to respond to these challenges, and to meet relevant user needs, the European ManyLaws project [3-4] has developed a suite of user-centric services that will ensure the real-time provision and visualization of cross-country and multi-lingual legal information to citizens, lawyers, businesses and administrations, as well as advanced analyses of it (including comparisons between countries, time-wise evolutions, identification of connections as well as conflicts among laws, transpositions of EU Directives, etc.). The proposed solution is based on a platform supported by the proper environment for semantically annotated Big Open Linked Legal Data (BOLLD). The ultimate objective of the project has been to provide the technical foundation and the tools for the development of a second-generation legal data e-infrastructure, making cross-country and multilingual legal information available to everybody, in a customizable, structured and easy-to-handle way, as well as all the required processing and analysis of it in order to become practically manageable and highly useful. Achieving this objective is particularly important in the European legal context, wherein multilingualism facilitates near-universal accessibility to different Member States' legal frameworks and thereby promoting greater European integration.

This research paper presents and validates a methodology for evaluating such ‘sec-
ond generation’ legal open data retrieval infrastructures based on the concept of ‘value
model’ estimations from users’ evaluation ratings [5-8]. This approach moves beyond
the traditional Information Systems (IS) evaluation approaches, as it not only includes
assessments of the magnitudes of a wide range of types of value generated by such an
infrastructure, but also considers the relations among them as well (see section 3 for
more details). This enables a deeper understanding of the whole value generation mech-
anism and offers a rational definition of priorities for improvements in the capabilities
offered to users. In particular, a multi-layer value flow model of such a ‘second gener-
ation' legal open data retrieval infrastructure has been developed, having as its theoret-
ical foundation the IS Success Model proposed by DeLone and McLean [9-12], sup-
ported by a methodology for estimating it using evaluation data collected from users.
The proposed methodology has been used for the evaluation of the second-generation
legal open data e-infrastructure developed in the abovementioned European project
ManyLaws. The research presented in this paper can be considered as very useful, since
second-generation open legal data e-infrastructures are quite new - still in their infancy
- and are characterized by important novelties and innovations. An extensive and de-
tailed evaluation of them is, therefore, required in order to assess the value they really
generate, to identify their strengths as well as weaknesses, and to define required im-
provements using advanced IS evaluation approaches in order that they may evolve
towards higher levels of maturity.

The research paper is structured as follows. In Section 2, the background of the pro-
posed methodology is delineated. The above-mentioned evaluation methodology is
then described in Section 3. Next, in Section 4, the second-generation legal open data
e-infrastructure developed in the European project ManyLaws is outlined. The applica-
tion of the proposed evaluation methodology to this e-infrastructure is presented in Sec-
tion 5. Finally, in Section 6, the conclusions are summarized, and future research direc-
tions are proposed.

## 2 Background

Extensive research has been conducted on information systems evaluation [13-19], a significant proportion of which has concluded that IS evaluation is a difficult and complex task. This is because there exist many different types of IS, each having different objectives, and aiming at different types of benefits, both financial and non-financial, and also tangible and intangible. Thus, the assessment of each particular type of IS requires a different evaluation methodology, which takes into account its particular objectives as well as capabilities. In [13] two basic directions of IS evaluation are identified: (a) the ‘efficiency-oriented’ direction, which evaluates IS performance with respect to some predefined technical and functional specifications, focusing on answering the question of whether the IS ‘is doing things right’; and (b) the ‘effectiveness-oriented’ direction, that evaluates the extent to which the IS supports the execution of business-level tasks or the achievement of business-level objectives, and focuses on answering the question of whether the IS ‘is doing the right things’. [19] conducts a review of previous literature on the evaluation of IS. The authors conclude that IS evaluation methodologies focus mainly on the ‘goal’, ‘environment’ and ‘activity’ aspects of the evaluated IS: they evaluate mainly to what extent the IS contributes to the attainment of business goals, is useful to employees (who constitute that most important part of its environment, corresponding to IS ‘effectiveness’), and also meets high levels of performance and accuracy (which correspond to IS ‘efficiency’) respectively.

Furthermore, extensive research in this area has been conducted on IS success [9-12], which has identified several dimensions/measures of it. The most widely used IS success model has been developed by DeLone and McLean [9-10, 11]. The initial
model identifies six IS success dimensions/measures, structured in three layers: (i) ‘information quality’ and ‘system quality’ (first layer); (ii) which affect ‘user satisfaction’ and also the ‘actual use’ of the IS (second layer); (iii) these two variables determine the ‘individual impact’ and the ‘organizational impact’ of the IS (third layer) [9, 11]. Subsequently, an updated version of the model was developed [10, 11], based on the experience gained from its extensive use; one which defines the following six dimensions of the success of an IS: (i) ‘system quality’, (ii) ‘information quality’ and (iii) ‘service quality’ (at the first layer), which affect (iv) ‘user satisfaction’ and the actual (v) ‘use’ or ‘intention to use’ (at the second level), and these affect (vi) the ‘net benefits’ that the IS generates (at the third layer). In [12] a re-specification and extension of this model is proposed, which includes perceived usefulness instead of actual use.

Based on a synthesis of the main conclusions of the IS evaluation research stream (briefly reviewed in the first paragraph of this section) on the one hand, and on the other hand the IS success research stream (briefly reviewed in the second paragraph of this section), the ‘value model’-oriented IS evaluation approach has been developed [5-8]. It consists of two stages:

i) Specification of a two layered ‘value model’ of the specific IS under evaluation. The first layer includes ‘efficiency-oriented’ value measures, which are concern with the main capabilities provided the IS, as well as its technical quality (e.g., availability, response time, etc.) and usability (i.e., properties of the IS in which we can directly intervene). The second layer includes ‘effectiveness-oriented’ value measures, which consider the extent of support provided by the IS for the accomplishment of users’ business-level objectives (i.e., the specific objectives concerning their tasks that the users want to achieve using the IS. For example. for an internal IS of a government agency, the main business objective of the public servants using it is to increase their working efficiency, while for a legal information e-infrastructure the main business objective of the lawyers using it is to improve their productivity and the quality of their performance of legal tasks), as well as the extent of use of it and overall satisfaction from it (i.e. properties of the IS in which we cannot directly intervene, but result from the first layer properties). The value model specification also includes interconnections/relations between first layer value measures and second layer ones (i.e. quantifying impacts of the former on the latter).

ii) Estimation of the above value model using evaluation data collected from users of this IS (e.g., through a questionnaire). This dimension includes a) calculation of the average ratings of the value measures of the first and the second layer; b) estimation of the above interconnections/relations between first layer value measures and second layer ones; this can be done either through the estimation of regression models, having as dependent variables the second layer value measures, and independent ones the first layer value measures, or in case of high correlations among the latter (multi-collinearity problems [20]), which is usually the case, we can calculate the correlations of the first layer value measures with the second layer ones. Based on the above results it is possible to identify strengths and weaknesses of the IS (= value measures that have received high and low user ratings respectively); and also, identify improvement priorities (= first layer value measures that have received low user ratings, and at the same time have
high impact on the second layer value measures, that is, the extent to which business-level objectives have been accomplished, the overall user satisfaction, etc.

## 3 Evaluation Methodology

A methodology for evaluating ‘second generation’ big open linked legal data (BOLLD) e-infrastructures has been developed, based on the abovementioned ‘value model’-oriented IS evaluation approach. Initially a value model of such an e-infrastructure has been specified, having the two-layered structure described above, elaborated using the IS Success Model of DeLone and McLean [9-12] as theoretical foundation, and also the novel capabilities offered by these ‘second generation’ BOLLD e-infrastructures. This is shown below, in Fig.1. It can be seen that the model includes five first layer groups of value measures, which can be viewed as ‘value dimensions’: three of them map to the ‘system quality’ dimension of the DeLone and McLean IS success model (capabilities, or functionalities provided by the e-infrastructure, including both the ‘traditional’ and the novel-innovative ones, ease of use-usability, technical quality), one concerning the ‘information quality’ and another one concerning ‘service quality’.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Value model of a second-generation big open linked legal data e-infrastructures</figcaption></figure>

The model also includes two second layer groups of value measures that are concerned with the extent of the support provided for the accomplishment of users’ business-level objectives (e.g. performing various legal tasks), which is an adaptation of the ‘net benefits’ dimension of the DeLone and McLean IS success model, as well as the user’s
overall satisfaction (which is a combination of the ‘user satisfaction’ and the ‘intention to use’ dimensions, as the former).

For each of the above seven value dimensions individual value measures are defined taking into account, on one hand, the items developed for measuring the main IS success dimensions defined by relevant models (see [11] for a comprehensive review), and, on the other, the capabilities (both the traditional and the novel ones) offered by the specific BOLLD e-infrastructure. Based on these value measures, a questionnaire has been developed for collecting users’ evaluation data for them. Applying these data, the value model of the e-infrastructure has been estimated based on the following algorithm that consists of the following six steps:

a) For each value dimension an aggregate variable is calculated as the average of its individual value measures' variables.

b) Average ratings are calculated for all value measures and dimensions (using for the latter the aggregate variables calculated in the previous step). This allows us to identify ‘strengths’ and ‘weaknesses’ of the BOLLD e-infrastructure.

c) For each aggregate variable of the second layer assessing one of the ‘dependent’ value dimensions, we estimate a regression having it as dependent variable, and having as independent variables all the aggregate variables of the first layers in order to estimate to what extent this value dimension is affected by value dimensions of the first layer. This is quantified by the R² coefficient of the regression [20].

d) For each value dimension of the first level we calculate its impact on the higher-level value dimensions (of the second layer), using again the aggregate variables calculated in step b. For this purpose, we can use the corresponding standardized coefficients of the regressions of the above step c, or (in case of multi-collinearity problems, which is usually the case) the correlations between the first layer of value dimension variables with second layer ones.

e) By combining the average ratings calculated in step b with the correlations calculated in step d we can construct a ‘high level’ value model of the BOLLD e-infrastructure at the level of value dimensions, and also a more detailed ‘low level’ one at the level of value measures. These value models enable a deeper understanding of the whole value generation mechanism of the BOLLD e-infrastructure, and we can also provide a colored intuitive visualization of these value models using ‘hot colors’ (e.g., red) for strong connections of first layer value dimensions/measures with second layer ones (e.g., having correlation higher than 0.6), and ‘cold colors’ for weaker connections.

f) Finally, the value dimensions and the value measures of the first layer, which are the only ‘independent variables’ within the control of the BOLLD e-infrastructure developer are classified, based on their average ratings by users and their average impacts on the value dimensions of the second layer ones, into four groups: low rating – high impact, low rating – low impact, high rating – high impact and high rating – low impact. High priority for improvement should be assigned to the improvement of first group of value dimensions and measures as they receive low evaluations by the users, and at the same time have strong impact on the generation of higher-level value concerning the accomplishment of users’ legislation-related objectives and overall satisfaction.
## 4 A Cross-border Big, Open and Linked Legal Data e-Infrastructure

In this section the main novel capabilities offered by the cross-border BOLLD e-infrastructure developed in the European ManyLaws project are briefly outlined (more detailed information on it are provided in [3-4]):

* Parallel search in multiple EU member-state legal frameworks (including European legislation or EU directives); this process will be effectuated through the parallel translation of queried search terms, using a suitable legal vocabulary.

* Interrelation of laws and news or social media posts, using a sentiment analysis; this service will permit users to stay informed about ongoing policy trends, as well as public opinion related to the creation of new laws or the review of existing ones.

* Comparative analysis of related/connected laws from the same national legal framework; this is presented as a text visualization and will give the user the ability to identify correlations, dependencies and conflicts between different laws.

* Timeline analysis for all legal elements; this functionality provides a visualization of the progress and current status of a specific piece of national legislation (after amendment/extensions) over time, including preparatory acts and agreements.

* Visualization of the connection between an EU directive and a national legal framework. This visualization will be presented through the system as a graph, wherein the connection would be clearly identified. This functionality will allow the user to assess the degree of transposition of an EU Directive into national law.

## 5 Application

The proposed evaluation methodology, which has been described in section 3, has been applied for the evaluation of the first version of this second-generation BOLLD e-infrastructure under development in the abovementioned project ManyLaws.

### 5.1 Value Model Specification

Initially a value model has been specified for the ManyLaws BOLLD e-infrastructure, based on the general specification shown in Fig. 1. This model includes all the value dimensions that the latter proposes, with the only exception of the service quality value dimension, This aspect was not included, since the e-infrastructure was not in production mode, and there were consequently no support services available to its users. Our value model is shown later in Fig.2. It is worth noting that it includes two technical quality value dimensions pertaining to the performance and the availability of the e-infrastructure. The value measures of each of the seven value dimensions are outlined below, in Table 1. The ‘Capabilities’ and ‘Objectives Accomplishment’ value dimensions were based on the specific capabilities and objectives of the ManyLaws BOLLD e-infrastructure, while the remaining value dimensions were based on previous empirical research using the DeLone and McLean IS Success model [9-11].
## 5.2 Data Collection

A series of workshops were held as part of the ManyLaws project, during and after the testing period in order to demonstrate the functionalities of the BOLLD e-infrastructure developed, to raise public awareness of its existence, and to allow for widespread testing of the prototype. Out of a pool of 100 potential volunteers (from the three main target groups of this e-infrastructure: legal professionals, legal researchers and public servants), 42 individuals responded and committed to pilot testing the system and implementing some predefined scenario with it. For the purposes of monitoring and evaluation within the purview of the ManyLaws project, a self-administered questionnaire was used to collect primary data from those individuals who had participated in the evaluation tests and associated workshops. The questionnaire included a series of questions that corresponded to the value measures of the abovementioned seven-value dimensions (see Table 1). These questions took the form of statements concerning some aspect of the BOLLD e-infrastructure (e.g., 'The interface of the system was pleasant and easy to look at'), and the users were asked to indicate the extent of their agreement or disagreement with each of them, using a five-point Likert scale (1=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree). In order to maximize coverage within the shortest period of time, it was decided to make use of an online questionnaire to collect information. The questionnaire was developed and hosted on Google Forms.

These evaluation data collected from the above respondents, after being processed using the methodology described in Section 3, resulted finally in the construction of the value model of this novel second generation BOLLD e-infrastructure. These data will provide a first understanding of users' perceptions and assessments concerning its main characteristics and functional capabilities (first layer value), as well as the extent of support it provides for performing important legal tasks and the overall satisfaction from it (second layer value). This will allow for the identification of both strengths and weaknesses from various perspectives. Further, it will enable an initial understanding of the different importance of these first layer characteristics and functional capabilities for generating higher layer value (support for performing important legal tasks, of both national and international scope, as well as overall satisfaction), and also of priorities for improvements of these characteristics and functional capabilities.

With respect to the demographics of the 42 pilot users and respondents of the questionnaire, they could be characterized as being legal experts: most of them were public servants (46.7%), followed by legal professionals (22.2%) and then legal researchers (17.8%). A small number of participants self-identified as being businesspersons. In terms of age, 46.7% of the respondents were in the group of 45-54 years old, while 31.1% was above 75 years old. The rest vary from 18 to 74 years old. Roughly two-thirds (64.4%) of all respondents described themselves as advanced ICT users, while 26.7% self-identified as being intermediate users. Eighty percent of the pilot users stated that they used legal data primarily for professional purposes, which they find mostly online (40%) or both online and offline (60%). Finally, a large proportion of respondents reported that they spent roughly 30 minutes per day searching for legal
information (42.2%); while others reported spending either 1-2 hours (28.9%), or more than 3 hours (28.9%), on the same task.

## 5.3 Data Analysis - Value Flow Estimation

In Table 1, below, the average rating for all value measures and dimensions (results for value dimensions in bold) are shown in the second column. With respect to the first layer value dimensions, it may be seen that the two technical quality related ones, availability and performance, are assessed as high (average ratings 4.14 and 4.05 respectively), while the remaining three - capabilities, usability and information quality – are perceived as moderate to high (average ratings 3.74, 3.73 and 3.52). Availability has received the highest average rating (4.14), so it can be considered as a strength of the e-infrastructure, while the information quality has received the lowest average rating (3.52), so it constitutes a weakness. With respect to the second layer value dimensions, we can see that both are assessed as moderate to high (average ratings 3.75 and 3.48). Proceeding to a higher level of detail, for the first layer individual value measures, we can see that most of them are assessed between moderate to high (average rating 3.5) and high (average rating 4.0). The accessibility of the platform using any browser, the availability anytime and from anywhere, and the capabilities of retrieving laws and legal documents in general have received the highest average ratings (exceeding 4.00) (strengths). In juxtaposition, the lowest average ratings have been given to the provision of only relevant results, the assessment of conflicts, comparisons or dependencies between different laws inaccuracies and the manual annotation of text (below 3.50) (weaknesses). For the second layer value measures, it may be seen that the ones concerning accomplishment of legal tasks related objectives are assessed between moderate to high (average rating 3.5) and high (average rating 4.0). For the value measures concerned with overall satisfaction with the e-infrastructure, we can see that the one associated with future use of it has received the highest average rating (between high and very high: 4.26), the one associated with paying a fee has received the lowest (less than moderate: 2.79), while the other two received average ratings between moderate and high.

<table><caption>**Table 1.** Average ratings and correlations of value dimensions and measures</caption><thead><tr><th>Measure/ Dimension</th><th>Description</th><th>Average ratings</th><th>Correlation OBJAC</th><th>Correlation SAT</th><th>Average Correlation</th></tr></thead><tbody><tr><td>CAP</td><td></td><td>3.74</td><td>0.824**</td><td>0.775**</td><td>0.800</td></tr><tr><td>CAP1</td><td>Search for legal information on a particular topic in different EU Member States' legislations</td><td>3.86</td><td>0.773**</td><td>0.700**</td><td>0.736</td></tr><tr><td>CAP2</td><td>Retrieve a particular law or legal document</td><td>4.02</td><td>0.488**</td><td>0.425**</td><td>0.456</td></tr><tr><td>CAP3</td><td>Access accurate translations of a law or legal document in my language</td><td>3.90</td><td>0.718**</td><td>0.571**</td><td>0.644</td></tr><tr><td>CAP4</td><td>Compare laws on the same subject within the same country</td><td>3.81</td><td>0.732**</td><td>0.783**</td><td>0.757</td></tr></tbody></table>
<table><tr><td>CAP5</td><td>Compare laws on the same subject between different countries</td><td>3.71</td><td>0.676**</td><td>0.677**</td><td>0.676</td></tr><tr><td>CAP6</td><td>Assess the degree of transposition of EU directives into national legislation</td><td>3.79</td><td>0.718**</td><td>0.624**</td><td>0.671</td></tr><tr><td>CAP7</td><td>Assess the conflicts, comparisons or dependencies between different laws</td><td>3.45</td><td>0.583**</td><td>0.613**</td><td>0.598</td></tr><tr><td>CAP8</td><td>Trace the evolution of a piece of legislation over time</td><td>3.86</td><td>0.641**</td><td>0.592**</td><td>0.616</td></tr><tr><td>CAP9</td><td>Access highly informative visualizations depicting the above comparisons and contrasts</td><td>3.93</td><td>0.736**</td><td>0.572**</td><td>0.654</td></tr><tr><td>CAP10</td><td>Access different types of parliamentary data</td><td>3.74</td><td>0.635**</td><td>0.684**</td><td>0.659</td></tr><tr><td>CAP11</td><td>Report inaccuracies and manually annotate text</td><td>3.36</td><td>0.628**</td><td>0.642**</td><td>0.635</td></tr><tr><td>CAP12</td><td>Access relevant public opinion data</td><td>3.48</td><td>0.685**</td><td>0.658**</td><td>0.671</td></tr><tr><td>USAB</td><td></td><td><b>3.73</b></td><td><b>0.776**</b></td><td><b>0.829**</b></td><td><b>0.802</b></td></tr><tr><td>USAB1</td><td>It was easy to find the information I needed</td><td>3.64</td><td>0.545**</td><td>0.620**</td><td>0.582</td></tr><tr><td>USAB2</td><td>The interface of the system was pleasant and easy to look at</td><td>3.88</td><td>0.681**</td><td>0.645**</td><td>0.663</td></tr><tr><td>USAB3</td><td>The output/results it provides are understandable</td><td>3.81</td><td>0.749**</td><td>0.701**</td><td>0.725</td></tr><tr><td>USAB4</td><td>The capabilities provided by the system are compliant with the work-practices and the mentality of legal professionals</td><td>3.59</td><td>0.527**</td><td>0.696**</td><td>0.611</td></tr><tr><td>PERF</td><td></td><td><b>4.05</b></td><td><b>0.122</b></td><td><b>0.230</b></td><td><b>0.176</b></td></tr><tr><td>PERF1</td><td>The system returned rapid results to my queries</td><td>3.86</td><td>0.016</td><td>0.150</td><td>0.083</td></tr><tr><td>PERF2</td><td>The speed at which the system returned results remained consistent for each login session</td><td>3.74</td><td>0.065</td><td>0.140</td><td>0.102</td></tr><tr><td>PERF3</td><td>The pages work in my favourite browser(s)</td><td>4.55</td><td>0.291*</td><td>0.315*</td><td>0.303</td></tr><tr><td>AVAIL</td><td></td><td><b>4.14</b></td><td><b>0.333*</b></td><td><b>0.253*</b></td><td><b>0.293</b></td></tr><tr><td>AVAIL1</td><td>I was able to access and browse the platform at my convenience - at any time of the day, from anywhere</td><td>4.12</td><td>0.325*</td><td>0.274*</td><td>0.299</td></tr><tr><td>AVAIL2</td><td>I was able to access and navigate through the different services at my convenience - at any time of the day, from anywhere</td><td>4.07</td><td>0.292*</td><td>0.226*</td><td>0.259</td></tr><tr><td>AVAIL3</td><td>The platform was never offline at the moment that I wanted to use it</td><td>4.24</td><td>0.289*</td><td>0.186*</td><td>0.237</td></tr><tr><td>INFQ</td><td></td><td><b>3.52</b></td><td><b>0.679**</b></td><td><b>0.670**</b></td><td><b>0.674</b></td></tr><tr><td>INFQ1</td><td>The results returned by the system correspond closely to the corresponding queries</td><td>3.57</td><td>0.649**</td><td>0.640**</td><td>0.644</td></tr><tr><td>INFQ2</td><td>The proportion of non-relevant results to my queries provided by the system is low</td><td>3.26</td><td>0.353**</td><td>0.408**</td><td>0.380</td></tr><tr><td>INFQ3</td><td>The system is able to recognize different keywords from the same legal domain</td><td>3.50</td><td>0.613**</td><td>0.578**</td><td>0.595</td></tr><tr><td>INFQ4</td><td>The translations made by the system are reasonably accurate</td><td>3.74</td><td>0.598**</td><td>0.557**</td><td>0.577</td></tr><tr><td>OBJAC</td><td></td><td><b>3.75</b></td><td></td><td></td><td></td></tr><tr><td>OBJAC1</td><td>To gain a better picture/understanding of the existing legislation on a particular topic in your country</td><td>3.83</td><td></td><td></td><td></td></tr></table>
<table><tr><td>OBJAC2</td><td>To gain a better picture/understanding of the existing legislation on a specific topic in other EU Member States and also at European Directives level</td><td>3.76</td><td></td><td></td></tr><tr><td>OBJAC3</td><td>To increase your productivity in performing various legal tasks involving legislation of your country</td><td>3.74</td><td></td><td></td></tr><tr><td>OBJAC4</td><td>To increase your productivity in performing various legal tasks involving legislation of other EU Member States and also legislation at European Directives level</td><td>3.83</td><td></td><td></td></tr><tr><td>OBJAC5</td><td>To improve the quality of performing various legal tasks involving legislation of your country</td><td>3.62</td><td></td><td></td></tr><tr><td>OBJAC6</td><td>To improve the quality of performing various legal tasks involving legislation of other EU Member States and also legislation at European Directives level</td><td>3.74</td><td></td><td></td></tr><tr><td>SAT</td><td></td><td><b>3.48</b></td><td></td><td></td></tr><tr><td>SAT1</td><td>I am confident that the system compares favorably with other available, similar legal informatics solutions</td><td>3.52</td><td></td><td></td></tr><tr><td>SAT2</td><td>I would like to use the system again</td><td>4.26</td><td></td><td></td></tr><tr><td>SAT3</td><td>I would be willing to pay a subscription fee to use the system again</td><td>2.79</td><td></td><td></td></tr><tr><td>SAT4</td><td>I would choose this system over other similar legal informatics products</td><td>3.33</td><td></td><td></td></tr></table>

As a next step we examined the extent to which the two value dimensions of the second layer are affected by the ones of the first layer (step 4). For this purpose, we estimated two regression models having as dependent variables the two value dimensions of the second layer Objectives Accomplishment (OBJAC) and Satisfaction (SAT), and as independent variables the five value dimensions of the first layer. The R² coefficients of these two regression models are 0.742 and 0.725 respectively, indicating that both second layer value dimensions are affected to a large extent by the ones of the first layer.

Finally, we calculated the correlations of the first layer value dimensions, as well as their value measures, with the two value dimensions of the second layer OBJAC and SAT. The results are shown in the third, fourth and fifth column of Table 1. In the fifth column, the average of these two correlations is demonstrated. We remark that the capabilities and usability, followed by the information quality, have high correlations with the two second layer value dimensions, and hence they have strong impacts on higher level value generation. The availability dimension has a moderate correlation with the two second layer value dimensions, thus its impact on higher level value generation is moderate, while the correlations of the performance dimension is not statistically significant.

Using the average ratings and correlations shown in Table 1 we can construct the value model of the BOLLD e-infrastructure; both the ‘high level’ one at the level of value dimensions, and the more detailed ‘low level’ one at the level of value measures. The former is illustrated in Fig.2. Our model provides a compact visualization of the main dimensions/types of value generated by the e-infrastructure (quantified through the corresponding average users’ ratings), the relations among them (quantified through
the corresponding correlations), and the main value generation paths. This enables a better understanding of the value generation mechanism of the BOLLD e-infrastructure under consideration.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Value model of the ManyLaws BOLLD e-infrastructures</figcaption></figure>

Based on these average ratings, and correlations presented in Table 1, improvement priorities were identified. At the level of first layer value dimensions, we remark that in terms of average rating two groups may be distinguished: a higher average rating one, consisting of the technical quality value dimensions PERF and AVAIL, and a lower average rating one, consisting of CAP, USAB and INFQ. Furthermore, in terms of correlations with second layer value dimensions, we can distinguish two groups of first layer value dimensions: one with higher correlations, consisting of CAP, USAB and INFQ, and one with lower correlations, consisting of PERF and AVAIL. Hence, it may be concluded that our highest priority should be given to the improvement of the CAP (capabilities), USAB (usability) and INFQ (information quality), as they have received lower ratings by the users, and at the same time have higher impact on higher level value generation concerning accomplishment of users’ legislation-related objectives and overall satisfaction, especially to the information quality.

Similarly, we have identified improvement priorities at the more detailed level of value measures. In particular, we have identified 6 out of the 26 value measures of the first layer, which belong to the lower 50% (= bottom 13) in terms of average rating, and to the upper 50% (= top 13) in terms of average correlation with the two second layer value dimensions, which should be assigned improvement priority:
CAP5: Compare laws on the same subject between different countries.

CAP6: Assess the degree of transposition of EU directives into national legislation.

CAP10: Access different types of parliamentary data.

CAP11: Report inaccuracies and manually annotate text.

CAP12: Access relevant public opinion data.

INFQ1: The results returned by the system correspond closely to my queries.

# 6 Conclusion

Unhindered access to legislation produced at different levels of government is important for the pursuance of economic and political activities. This recognition has led to the development of a ‘first-generation’ of legal open data e-infrastructures, mainly at the national level. Rapid globalization, the emergence of supra-national entities such as the European Union, and the ever-increasing volume, complexity and dynamism of national and international legislation has necessitated the development of a ‘second generation’ of cross-country BOLLD e-infrastructures that offer a much more extensive legal information than the first-generation ones. It is recognized that this new iteration of legal infrastructures needs also to offer the user more sophisticated processing and analytical capabilities to handle of this massive legal information, in order to make it manageable and more practically useful.

In previous sections of this research paper, a methodology for evaluating the emerging second generation of BOLLD e-infrastructures has been presented, adopting an advanced value model oriented IS evaluation approach. Our model is based on the estimation of value models of these BOLLD e-infrastructures, which include assessments of the main types of value they generate, and also the relations among them. The proposed approach enables not only the identification of strengths and weaknesses, but also a deeper understanding of the value generation mechanism and a rational definition of improvement priorities. It should be noted that it can be used, with some adaptations, for the evaluation of the ‘traditional’ first generation BOLLD infrastructures as well, and also for future more advanced second-generation ones, contributing to the evolution and maturity of legal information provision for supporting legal tasks of lawyers, public servants, business and even individual citizens. A first application of this approach was made for the evaluation of an advanced second generation BOLLD e-infrastructure developed under the aegis of the European project ManyLaws, leading to interesting insights, as well as improvement priorities. The present study also makes a significant contribution to existing literature, and that body of knowledge concerned with the value model construction approach to IS evaluation.

More research is required relating to the further application of the proposed methodology for the evaluation of next versions of the same BOLLD e-infrastructure, and also of other advanced second generation BOLLD infrastructures, based on larger ‘professional’ users’ groups. There also remains scope for further enquiry into the extension of this BOLLD e-infrastructures’ evaluation methodology, with the potential addition of more first and second layer value dimensions and value measures.
References

1. European Union (2020). Shaping Europe's Digital Future – Factsheet. https://ec.europa.eu/commission/presscorner/detail/en/fs_20_278.

2. Virkar, S., Alexopoulos, C., Stavropoulou, S., Tsekeridou, S., Novak, A., Charalabidis, Y. (2020). User-Centric Decision Support System Design in Legal Informatics: A Typology of Users. In: Charalabidis, Y., Cunha, M.A., Sarantis, D. (eds.) Proceedings – ICEGOV 2020 13th International Conference on Theory and Practice of Electronic Governance, 23-25 September 2020, Athens, Greece, pp.711 – 722. ACM Press.

3. Charalabidis, Y., Loutsaris, M.A., Virkar, S., Alexopoulos, C., Novak, A.-S., Lachana, Z. (2019). Use case scenarios on legal machine learning. In: Ben Dhaou, S., Carter, L., Gregory, M. (eds.) Proceedings - ICEGOV 2019 12th International Conference on Theory and Practice of Electronic Governance, 3-5 April 2019, Melbourne, VIC, Australia, pp 364 – 373. ACM Press (2019).

4. Virkar, S., Tsekeridou, S., Romas, I., Alexopoulos, C.: Towards Comprehensive Legal Information Search and Retrieval: Learning and Building from Evaluated Projects (2020). In: Proceedings - Internationales Rechtsinformatik Symposion - IRIS 2020, 27 – 29 February 2020, Salzburg, Austria (2020).

5. Pazalos, K., Loukis, E., Nikolopoulos, V. (2012). A Structured Methodology for Assessing and Improving e-Services in Digital Cities. Telematics and Informatics (SCOPUS), Vol. 29, pp. 123-136.

6. Loukis, E. Pazalos, K. Salagara, A. (2012), “Transforming e-services evaluation data into business analytics using value models”, Electronic Commerce Research and Applications, 11(2), 129–141.

7. Charalabidis, Y., Loukis, E., & Alexopoulos, C. (2014). Evaluating second generation open government data infrastructures using value models. In 47th Hawaii International Conference on System Sciences (HICSS). (pp. 2114-2126).

8. Loukis, E. and Leou, T. (2019). Evaluating Complex Government SAAS Through Value Flow Model Estimation. In Proceedings of 13th Mediterranean Conference on Information Systems (MCIS 2019), Naples, Italy.

9. DeLone, D. H., McLean, E. R. (1992). Information Systems Success: The Quest for the Dependent Variable. Information Systems Research, 3(1), pp. 60-95.

10. DeLone, D.H., McLean, E. R. (2003). The DeLone and McLean model of information systems success: a ten-year update. Journal of Management Information Systems, 19(4), 9-30.

11. DeLone, W. and McLean, E. (2016). Information Systems Success Measurement. Foundations and Trends in Information Systems, 2(1), 1–116.

12. Seddon, P. B. (1997), A respecification and extension of the DeLone and McLean model of IS success. Information Systems Research, 8(3), 240–253.

13. Smithson, S., and Hirscheim, R. (1998). Analysing information systems evaluation: another look at an old problem. European Journal of Information Systems, 7, 158-174.

14. Farbey, B., Land, F., and Targett, D. (1999), "Moving IS evaluation forward: learning themes and research issues. The Journal of Strategic Information Systems, 8(2), 189-207.

15. Willcocks, L., and Graeser, V. (2001), "Delivering IT and E-business Value". Butterworth-Heinemann, Boston, MA.

16. Irani, Z. (2002). Information systems evaluation: navigating through the problem domain. Information and Management, 40, 1, 11-24.

17. Gunasekaran, A., Ngai, E. W. T., and McGaughey (2006). Information technology and systems justification: a review for research and applications. European Journal of Operational Research, 173, 957-983.
18. Irani, Z., and Love, P., (2008), "Information systems evaluation - a crisis of understanding" In Z. Irani and P. Love (eds.), "Evaluating Information Systems - Public and Private Sector', Butterworth-Heinemann, UK.

19. Prat, N., Comyn-Wattiau, I. and Akoka J. (2015) A Taxonomy of Evaluation Methods for Information Systems Artifacts. Journal of Management Information Systems, 32(3), 229-267.

20. Greene, W. H. (2011), 'Econometric Analysis - 7th edition', Prentice Hall Inc., Upper Saddle River, New Jersey.