A qualitative case study of the adoption and use of an agricultural decision support system in the Australian cotton industry: The socio-technical view In response to the call for research that considers the human as well as the technical aspects of information systems implementation, the authors report on an interpretive case study which explores the adoption and use of an agricultural decision support system (DSS) CottonLOGIC in the Australian cotton industry. The study was informed through the innovation-decision model by Rogers and the technology-in-practice model by Orlikowski using a socio-technical approach. It was found that participants who achieved a high level of implementation success were reflexive and resourceful in adapting the technology to their changing needs, often in ways unanticipated by DSS builders. Australian farmers are supplementing traditional practices with innovative management strategies in an effort to survive recent economic and environmental crises in the rural sector. Cotton has become Australia's fourth largest agricultural export despite only about 1,500 cotton farms in Australia. The area under cotton is estimated at around 500,000 hectares although this area has diminished in the past year due to the effects of a severe and prolonged drought in Australia [39] .. These farms have diversified to grow other crops while many graze sheep and cattle [9] . Survival of the cotton industry is due in no small measure to the industry's commitment to research and practice.. While innovative technologies such as the computer-based agricultural decision support system (DSS). CottonLOGIC are considered influences on the adoption of sustainable farming systems in the cotton industry, the continued low adoption rates of agricultural DSS have been a concern [8, 10, 11, 15, 16, 17, 19, 21, 22, 23] .. DSS research has been critically evaluated in a recent, ongoing project by Arnott and Pervan [2, 3] who identified several key issues to be addressed. Some of the findings were: firstly, the relevance of DSS research would improve by increasing the number of case studies, especially interpretive case studies to help reduce the gap between research and practice; secondly, DSS empirical research is "overwhelmingly positivist, and more dominated by positivism than IS research in general" [2 (p.67)]; and finally, the academic rigour of DSS research would improve by broadening the reference theory foundation from its extant narrow base.. Furthermore, an alternative decision-making paradigm in DSS research to consider perspectives "beyond the technical" was proposed in a paper describing decision making in organisations by Courtney [7 (p.36) ]. According to Courtney [7] , the conventional view of DSS is both mechanistic and analytical in nature and "scarcely considers anything but the technical perspective" (p.30). Arnott and Pervan [2] extended the notion to allege that "a major omission in DSS scholarship is the poor identification of the clients and users of the various DSS applications that are the focus of investigation" (p.67). In response to the claims that social and personal views are neglected in DSS research, this paper takes a socio-technical approach to investigate the interaction of cotton growers on Australian family farms with the agricultural DSS CottonLOGIC by exploring both the societal and technical aspects of DSS implementation.. A multitude of definitions of computerised DSS exist. Turban, Aronson, Liang and Sharda [40] identified the functions of a DSS as an "adjunct to decision makers to extend their capabilities but not to replace their judgement" and to "support the solution of a certain problem or to evaluate an opportunity" (p.88). Arnott and Pervan [2] classified DSS as "the area of the information systems discipline focussed on supporting and improving managerial decision making" (p.67) while Hearn and Bange [16] , in their paper reviewing the role of DSS in the Australian cotton industry, claimed that DSS assist the decision making of crop managers while raising "awareness of other dimensions" (p.53). All these definitions characterise the agricultural DSS in this paper.. This paper revisits the data from an interpretive single case study in the Australian cotton industry which informed a doctoral thesis by the first author [24] . Aspects of CottonLOGIC implementation, in particular adoption and use, are expanded in this paper and analysed through the conceptual lens provided by two theories of sociological origins, outlined as follows. The adoption process of CottonLOGIC as innovative technology is examined through the perspective of the innovation-decision model from Rogers' diffusion of innovations theory [35] with its deterministic and objective focus on technology. More subjective and informal insights are elicited by applying Orlikowski's technologies-in-practice model [30] . This model is an extension of the duality of technology from Orlikowski's structurational model of technology [29] which is based on Giddens' structuration theory [12] . Essentially, the technologies-in-practice model offers a more human perspective to the study of technology 'in use' as distinct from technology as an artefact, an object, or an innovation, as conceived by Rogers [35] .. In brief, this paper highlights the value of considering both technical and social dimensions when investigating DSS implementation. The research objectives or problems of the study were formed through a critical review of the relevant literature, taking into account its ontological, methodological, theoretical, as well as practical goals. The problems to be addressed in this paper are summarised by the following research questions, with replies offered in the concluding section. The findings of this paper are expected to contribute to a greater appreciation of the issues facing cotton growers in their engagement with CottonLOGIC. While the contributions are outlined as follows, they are expanded more comprehensively in the relevant sections of this paper, and responded to in the concluding section. The ontological aim of the study is to provide a sociotechnical view of cotton growing and to contribute to scholarly knowledge in the understudied research domain of Australian cotton growers, agricultural DSS, and cotton farm management [17, 24] . The methodological significance stems from it interpretive nature since an inductive reading of the data provides in-depth insights into technology adoption and use, particularly from a client perspective. Moreover, studies suggest that relevance in DSS research increases through an empirical qualitative DSS case study [2, 3, 15, 21] . The conceptual framework used to inform the study is novel for agricultural DSS research with both innovation-decision and technologies-in-practice theoretical models having sociological origins. These theories offer increased academic rigour from a wider reference theory foundation for DSS research, which, in the past, has over-relied on decision-making theory [2, 3, 7, 33] . The practical implications of the study from the insights gained are expected to include the provisions of guiding principles for future agricultural DSS (11, 39] . The paper has the following structure. Firstly, an overview of the background literature relevant to DSS particularly in the rural sector is followed by a brief description of the research site and the research strategy of the study. Next is a review of the two theories which comprise the conceptual framework; then an analysis of the interview data through the sensitising concepts from each of the theories along with a discussion of the findings. Finally, is the concluding section where the findings are interpreted, responses to the research questions are given, and the contributions of the paper are discussed. The socio-technical approach of the paper has lent an alternative, more human perspective to the study of farm management using a technological artefact when, as McCown [23] asserted, nearly all DSS research in agriculture examines technological tools with little concern for the practicalities of decision support for farm management (p.20). Responses to the three research questions are given below.

O B B I O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O


Reconciling while tolerating disagreement in collaborative data sharing In many data sharing settings, such as within the biological and biomedical communities, global data consistency is not always attainable: different sites' data may be dirty, uncertain, or even controversial. Collaborators are willing to share their data, and in many cases they also want to selectively import data from others -but must occasionally diverge when they disagree about uncertain or controversial facts or values. For this reason, traditional data sharing and data integration approaches are not applicable, since they require a globally consistent data instance. Additionally, many of these approaches do not allow participants to make updates; if they do, concurrency control algorithms or inconsistency repair techniques must be used to ensure a consistent view of the data for all users.In this paper, we develop and present a fully decentralized model of collaborative data sharing, in which participants publish their data on an ad hoc basis and simultaneously reconcile updates with those published by others. Individual updates are associated with provenance information, and each participant accepts only updates with a sufficient authority ranking, meaning that each participant may have a different (though conceptually overlapping) data instance. We define a consistency semantics for database instances under this model of disagreement, present algorithms that perform reconciliation for distributed clusters of participants, and demonstrate their ability to handle typical update and conflict loads in settings involving the sharing of curated data. When multiple autonomous, collaborating parties agree to share data, they often encounter situations where that data is mutually inconsistent: each party has a data instance that is internally consistent, but the different parties may each have "facts" that conflict with the others. Such inconsistency arises from many reasons: sites may have different levels of data freshness, some sites' data may be dirty, or, frequently, the sites may have different viewpoints about what data is factual.. For instance, in bioinformatics, the same results from microarray experiments may be analyzed by different tools, or curated by different people, yielding different values. This is inevitable because most data curation is somewhat uncertain in nature, and thus correctness becomes a matter of interpretation. Therefore, bioinformatics data warehouse curators often reach different conclusions, and even occasionally revise these conclusions. Biologists have a sense of the authority of the different databases: for instance, SWISS-PROT [2] is generally more reliable than NCBI GenBank [24] because it is human-curated (see, e.g., [22] for a query language that incorporates such policies). To help control the quality of the data they use (and also to ensure their queries are not seen by competitors), most biologists create, query, and curate their own local database instances, which they populate with downloaded data. A biologist will typically not publish the contents of this database until he or she submits a paper for publication.. These situations result in a mismatch with existing methods of sharing data. If sharing is based on a common DBMS (whether distributed or client-server), the DBMS concurrency control and integrity checking routines will mandate that a single version be put into the database. Even in situations where local caching and optimistic concurrency control are used, ultimately one version of the state will exist. If, instead, the collaborating parties share information through a data integration [11, 20] or peer data management [13, 15] system, the results will be either inconsistent, or filtered through a "repair" scheme [1, 19] . Other strategies have similar drawbacks; e.g., file synchronization [26, 7] typically results in conflicts that must be manually resolved in order to restore consistency.. The solution to the needs of collaborating scientists is not to provide one globally consistent data instance, but rather to give each participant a custom, internally consistent instance with the data that this particular source accepts as authoritative. This instance should contain data "overlapping" with that at other sites, as well as certain data items that diverge from (many of) the others. In keeping with the needs of scientists as described above, data instances should be loosely coupled: each participant queries and manipulates a local database instance and may occasionally publish its database, including a log of recent changes; it may selec- Figure 1 : Collaborative data sharing system with three bioinformatics data warehouse participants sharing data on protein functions.. tively import others' modifications according to some policy. Today such capabilities can only be provided through custom import tools, which are expensive to develop and generally do not provide well-defined conflict resolution semantics.. The challenge in supporting this data sharing model lies in the fact that every database instance is subject to updates, in the form of transactions. Not all of these transactions may originate from sources trusted by all parties; hence, each site must decide whether to accept an update (in effect, synchronizing a portion of this site's data instance with the origin of the update), to reject it as being untrusted (causing a divergence), or in some cases to defer processing an update (if there are multiple conflicting updates with no clear preference). An update-centric model is important, as opposed to simply querying the resulting data values, because it allows sites to reject removals or replacements. However, it adds significant complexity, because one update may depend on a chain of other updates, some of which are trusted and others of which are not. A clean semantics for update propagation must consider these interactions, as well as different levels of trust.. At the University of Pennsylvania, we are developing a generic platform to support this type of data sharing; we refer to this platform as a collaborative data sharing system or CDSS. Our overall CDSS architecture was first presented in an overview paper [14] , which sketched out our vision of confederations of autonomous data sharers who publish and import updates. The process of selectively importing updates (in particular, those that do not conflict with existing data) is termed reconciliation. The Orchestra system represents our efforts to realize such a CDSS, and although our system is broadly applicable, we are particularly focusing on the needs of bioinformatics and biomedical researchers.. An Orchestra system ( Figure 1 ) consists of a number of collaborating participants (p1, . . . , p3 in the figure), each of whom controls and edits its own data instance (denoted I1, . . . , I3), and each of whom has a policy about what external data it is willing to trust and accept (labels on the arcs). As data is modified at different sites (denoted ∆1, . . . , ∆3), Orchestra publishes and propagates the updates to all sites that are willing to accept them. Our end goal is to facilitate update propagation in situations with both disparate schemas (i.e., each participant may have a different schema, with some attributes that may not have corresponding aspects in the other schemas) as well as disparate instances (i.e., each participant may have tuples with values that may not exist in other participants' instances). This paper represents our first step towards addressing these goals: it defines a model and methodology for reconciling instances of a single database schema, in the presence of transactions, disagreement, and trust relationships. The complementary problem of translating updates across schemas in order to incrementally maintain data instances is ongoing work.. We make the following contributions:. • A formalization for reconciliation that emphasizes local autonomy, making forward progress, and providing intuitive behavior. We support coexistence of multiple instances, with consistency defined by trust policies.. • A model for reconciliation where individual transactions are assigned trust levels, and the goal is to propagate the highest-trusted, longest chains of transactions.. • New algorithms to reconcile database instances while maintaining consistency .. • A performance analysis of these algorithms on bioinformatics-based workloads.. This paper is structured as follows. We discuss related work in distributed data sharing in Section 2. We describe our CDSS model in Section 3. In Section 4, we define a semantics for reconciliation. Section 5 presents algorithms for implementing reconciliation. We validate our initial prototype implementation in Section 6, and finally we conclude and discuss future work in Section 7. This paper represents a new means of providing distributed data sharing services. Rather than requiring all sites to agree on a single data instance that is globally consistent, instead we allow each site to have its own autonomousbut carefully coordinated -data instance, with a series of acceptance rules specifying its policy for importing updates from others. Our contributions have been the following:. • A formalism for reconciliation, including a notion of consistency, that emphasizes local autonomy, making forward progress, and providing intuitive behavior.. • New algorithms to reconcile database instances.. • Demonstration that these algorithms perform acceptably for sharing data in mid-sized confederations.. The work discussed in this paper represents a first step in the Orchestra project. In the future, we plan to address techniques for schema translation, implement a fully distributed network-centric reconciliation engine, and perform real-world evaluation of Orchestra in bioinformatics data sharing. We are optimistic that our approach of supporting overlapping, yet divergent, database instances will facilitate improved data sharing practices. The problem of sharing structured data among multiple sites has a long history, and it has been revisited in many different settings. However, as described in the previous section, there has generally been an assumption that the end goal is to create a single consistent data instance (perhaps with multiple views) across all sources. We briefly outline previous work.. File synchronizers [26, 10, 16] take two files, and they attempt to reconcile the changes into a maximally consistent instance. When a conflict exists, both versions are typically preserved and the user must resolve the conflict before continuing to work. The Bayou [9] disconnected file system provides mechanisms for supplying application-specific conflict detectors and conflict resolvers. The Coda [30] and Ivy [23] file systems provide a mechanism for detecting conflicts, and disable updates to them until a repair is made.. Distributed databases provide a single administrative domain, database schema, and instance to all users. Each participant may independently make edits to the database instance. Through either lock-based or optimistic concurrency control schemes [17, 8, 5] , the system interleaves updates while maintaining consistency. Version vectors [25] , vector clocks [18] and other mechanisms are often used to determine causality. An alternative approach is that of Lotus Notes [21] , which, upon a conflicting update, splits the data into two versions that are no longer synchronized.. Data integration (including federated databases and peer data management) is a middleware layer above autonomously maintained data sources. Here, each participant updates its own data sources in a self-contained fashion. When a participant wishes to access data from "the outside world," it poses a query over the virtual mediated schema that the data integration system builds over all data sources. In traditional data integration, there is one mediated schema; peer data management systems [13, 15, 4] allow for multiple mediated schemas, interrelated by a network of schema mappings. Data consistency is generally managed in terms of certain answers: inconsistent data values are removed through a "repair" procedure [1] . No facilities exist here for importing data into a given data source.. The collaborative data sharing system was first introduced in [14] , which proposed (but did not implement) an architecture for supporting the exchange of atomic updates in a distributed setting. Our work in this paper builds upon these ideas, but focuses on the complexities introduced by transactions and different trust levels.

O O O O O O B B O O B B O O O O O O O O B B B B O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O B B O B O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O B O O B B O O O B O B O O O O O O O O B O O O O O O B O O O O O O O B O O B O O O O B O O O B O O O O O O O B O O O O B O O B O O O B O O O O O O B O B O O O O B O O O O O O O B O O O O O B O O B O O O O O O O O B O O O O O O B O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O B O O O O B O O O O O O B O O O O O O O O O O O O B B O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O B O O O O B O B O O O O O O O O O O O O O B B O O O O B O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B B B O O O O O O B O O O O B O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O B B O O O O O O O O O O O O O B O O O O O O O O O O O B O O B O O B O O O O O B O O O O O O B O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O B O O O O O B O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O B O B O O B O O O O O O O O O O O O O O O B O O O O O B O O O B O O O O O O O O O O O O B O O O O O O O O O B B B O O O O O O O B O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O B B B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O B B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O B B O O O O O O O O O O O B O O O O B B O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O B O O O O O B O B O O O O O O B O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O B O O O O O O O O B O O O B B O O B O O O O O O O B O O O B O O O O O O O O O O O O O O O B O O O O O B O O O B O O O O O O O B O O O O O O O O O O O O O O O O O B B O O O O O O O O O B B O O O O O O O O O O O O O O B O O O B O O O B B O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O B O O O O O O O O O O O B O O O O


Using elimination to implement scalable and lock-free FIFO queues This paper shows for the first time that elimination, a scaling technique formerly applied only to counters and LIFO structures, can be applied to FIFO data structures, specifically, to linearizable FIFO queues. We show how to transform existing nonscalable FIFO queue implementations into scalable implementations using the elimination technique, while preserving lock-freedom and linearizablity.We apply our transformation to the FIFO queue algorithm of Michael and Scott, which is included in the Java TM Concurrency Package. Empirical evaluation on a state-ofthe-art CMT multiprocessor chip shows that by using elimination as a backoff technique for the Michael and Scott queue algorithm, we can achieve comparable performance at low loads, and improved scalability as load increases. Elimination is a parallelization technique that has shown promise in designing scalable shared counters [2, 20] and Last-In-First-Out (LIFO) structures such as pools and stacks [7, 20] . This paper shows the first example of applying elimination to First-In-First-Out (FIFO) structures, specifically, to one of the most fundamental and widely studied concurrent data structures in the literature: the concurrent FIFO queue [6, 9, 10, 12, 14, 17, 18, 19, 22, 23, 24, 25] .. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SPAA'05, July 18-20, 2005 , Las Vegas, Nevada, USA.. (c) Sun Microsystems, Inc. We have shown that, with care, the elimination technique already known to be useful for making stacks, pools and counters scalable, can also be applied to FIFO queues. Our preliminary performance results indicate that this technique improves performance even with relatively small numbers of concurrent threads and we are optimistic that the improvement due to elimination will only increase with larger numbers of concurrent threads.. Future work includes evaluating different heuristics for deciding whether to access the central queue or to eliminate in order to effectively adapt to a range of loads.

O B O O O O B B I O B O O O O O O B O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O B O O O O O O B O O O O O O O O B O O O B B O O O O O O O O O O O B O B I O O O O O O O O O O B O O O O O O O O O B B O O O O B O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O B O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O


Proof over promise: towards a more inclusive ranking of Dutch academics in Economics & Business Abstract The Dutch Economics top-40, based on publications in ISI listed journals, isto the best of our knowledge-the oldest ranking of individual academics in Economics and is well accepted in the Dutch academic community. However, this ranking is based on publication volume, rather than on the actual impact of the publications in question. This paper therefore uses two relatively new metrics, the citations per author per year (CAY) metric and the individual annual h-index (hIa) to provide two alternative, citation-based, rankings of Dutch academics in Economics & Business. As a data source, we use Google Scholar instead of ISI to provide a more comprehensive measure of impact, including citations to and from publications in non-ISI listed journals, books, working and conference papers. The resulting rankings are shown to be substantially different from the original ranking based on publications. Just like other research metrics, the CAY or hIa-index should never be used as the sole criterion to evaluate academics. However, we do argue that the hIa-index and the related CAY metric provide an important additional perspective over and above a ranking based on publications in high impact journals alone. Citationbased rankings are also shown to inject a higher level of diversity in terms of age, gender, discipline and academic affiliation and thus appear to be more inclusive of a wider range of scholarship. Economists love to rank. Even a casual five-minute literature search reveals literally hundreds of publications on rankings of academic productivity and impact. There are numerous rankings of universities (e.g., Kalaitzidakis et al. 2003) , departments (e.g. Scott and Mitias 1996; García et al. 2012) , journals (e.g. Kodrzycki and Yu 2006; Harzing and van der Wal 2009 ) and individuals (e.g. Tol 2009; Prathap 2010) . Dutch economists are no exception. In fact, they produced what, to our best knowledge, is the oldest ranking in the field: a nation-wide ranking of Economists (the Economics top-40) that has entered its fourth decade and is ''broadly accepted and supported by the Dutch academic community in the field'' (Nederhof 2008, p. 172) .. However, Nederhof (2008, p. 172) cautions us that the Economics top-40 is not based on actual impact and that ''overall the rankings induced economists to focus more on maximizing publication output than on optimalizing their citation impact''. Harzing (2005) reported a similar effect for Australia, where a government focus on output seemed to lead to a high volume/low impact publication profile for the field of Economics & Business as a whole. More recently, Franses (2014) published an analysis of three decades of the Dutch Economics ranking and suggested that looking at citations, instead of publications alone, would provide a useful alternative to the current ranking. We fully agree with this suggestion, as publication-even in so-called ''high impact'' journals-does not guarantee citation impact. We argue that rankings based on publications in high impact journals are merely reflecting the ''promise'' of academic impact, whereas only rankings based on citations in a wider range of academic outlets provide the actual ''proof'' for this.. Although this argument is not new, as far as we are aware there are no prior studies that explicitly compare publication-based with citation-based rankings. Given its long-established history the Dutch Economics top-40 provides an ideal test case for such a comparison. The remainder of this paper is structured as follows. We first discuss the history of the Dutch Economics top-40 and introduce citation-based rankings as an alternative to publication-based rankings. Subsequently, we present our research methods and results, comparing three different rankings of Dutch economists. A discussion section puts the results in a broader perspective. We suggest that citation-based rankings are more democratic and are likely to produce rankings that are more inclusive of a wider range of scholarship. This paper used two relatively new metrics, the CAY metric and the hIa. Which metric one prefers depends on the purpose of the investigation. The CAY is the age and authorcorrected equivalent of the total number of citations, whereas the hIa is the age and authorcorrected equivalent of the h-index. If one is most interested in the cumulative citation impact of an academic, the CAY is more appropriate. If one is most interested in the average number of high-impact publications an academics produces on a yearly basis, the hIa is more appropriate. As for other research metrics, the CAY or hIa-index should never be used as the sole criterion to evaluate academics. Another crucial question that should always be asked is: ''Has the scholar asked an important question and investigated it in such a way that it has the potential to advance societal understanding and well-being?'' (see e.g. Adler and Harzing 2009 ). However, we argue that the hIa-index and the related CAY metric provide an important additional perspective over and above a ranking based purely on publications in high impact journals alone. Citation-based rankings were also shown to inject a higher level of diversity in terms of age, gender, discipline and academic affiliation and thus appear to be more inclusive of a wider range of scholarship. This paper applied two fairly new metrics in a pilot study that created two new citationbased rankings for academics in Economics & Business in the Netherlands. Our new rankings were based on the ''proof over promise'' principle. Rather than simply inferring impact from publication in high impact journals, we measured actual impact through an investigation of article-level citations. We also used a more inclusive definition of research output, including not just articles in ISI-listed journals, but also books, book chapters, working and conference papers and articles in non-ISI listed journals. Although ISI listing is seen by many to imply a quality stamp, in our view it should not matter where research is published. If a particular research output is highly cited, it clearly influences the field and that should be more important than the journal in which it is published. Table 5 integrates the three rankings discussed in this article. As before, academics ranked on all three rankings are listed in green, academics that are ranked on both the Economics top-40 and the PoP CAY top-40 are listed in orange, academics ranked on both the Economics top-40 and the PoP hIa top-40 are listed in red, whereas academics ranked on both citation-based rankings are listed in blue. Academics in black are ranked on one list only. Table 5 also provides a comparison between the three rankings for each individual academic. Although for some academics the resulting range of rankings is relatively narrow, for others the three rankings provide widely diverging results. Correlations between the actual scores for the various metrics indicate that there is certainly some communality underlying the rankings. The scores used in the original Economics top-40 and the PoP CAY 2003-2013 metric have a correlation coefficient of 0.69. With a correlation coefficient of 0.44 the relationship between the Economics top-40 scores and the PoP career hIa metric is much weaker. This is not surprising as the hIa not only focuses on citations rather than publications, but also considers the entire career rather than just the most recent performance. The PoP CAY 2003-2103 and the whole of career hIa metrics show the strongest correlation at 0.79. Even so, the metrics are dissimilar enough to provide unique information, unlike most of the h-index variants that show correlation coefficients above 0.90 with the original h-index (see Bornmann et al. 2011; Harzing et al. 2014) .. We argue that our ''proof over promise'' approach is more ''democratic''/inclusive than the original Economics top-40. First, by expanding the type of research outputs considered beyond the narrow scope of publications in ISI-listed journals, we remove the disciplinary bias against Management, Marketing and Accounting & Finance, disciplines in which a smaller proportion of high-quality journals is ISI-listed than in Economics and Management Science (Harzing and van der Wal 2009 ). Whereas in the original publication-based Economics top-40 more than two thirds of the listed academics works in Economics or Management Science, in our citations-based rankings this proportion is reduced to just over half in the CAY ranking and to 43 % in the hIa ranking.. Second, citation-based performance metrics can be argued to be more democratic as their ''verdict'' is based on the reception of the paper by the academic community as a whole, whereas acceptance in a high-impact journal is dependent on only a handful of gatekeepers (the editor and reviewers). This also increases the chance that a publicationbased performance metric is influenced not just by the quality of the papers, but also by particularistic criteria such as reputation or personal networks of the author, the reputation of the university the author is affiliated with or its presence in the editorial board of journals. Although citations are certainly not immune to this mechanism, they are less sensitive to particularistic criteria and the effect might be mediated by journal prestige (Judge et al. 2007 ). Although we cannot establish with any certainty that these mechanisms are in operation, the citation-based rankings feature more academics from universities not traditionally seen as the primary research universities in Economics & Business in the Netherlands, such as Maastricht, Eindhoven and Nijmegen. They also include a larger number of women and younger academics. Third, our ranking was conducted with a free software program (Publish or Perish) and a publicly available data-base (Google Scholar). Hence, any reader can easily replicate the ranking without the need for subscription-based data-bases or complicated calculations based on percentile scores for the AIS. This also means that any academic can look up their own citation record and easily find out where they score in the current ranking. For instance, the first author of this article would rank 10th in a CAY-based ranking and 6th in a hIa-based ranking. When attempting to replicate our ranking, please note that Google Scholar is updated every couple of days, and that although one can limit the publication . Although the whole-of-career hIa will be relatively stable with only occasional increases when another publication enters the individual h-index, the CAY metric will continue to increase as the calendar year progresses and citations accrue. Both metrics will decline by definition after the start of a new calendar year as the ''academic age'' of the academic in question increases by 1 year.. Fourth, citation-based rankings and in particular the hIa ranking are likely to provide more dynamic rankings in terms of changes over the years. The original Economics top-40 is fairly stable over time, especially in the higher ranks. In an analysis of 29 years of Economics top-40 Franses (2014 Franses ( , p. 1268 indicates that: ''The main conclusion from this paper is that a small subset of all Dutch economists dominates the charts for years''. Once an academic supervises a large number of PhD students and is involved in a number of collaborative projects, publication output is likely to remain high, even if the resulting papers do not have a large citation impact. As in the Netherlands only full professors can be primary supervisors, this provides a built-in disadvantage for younger academics and those working in disciplines or universities that attract fewer PhD students. A ranking based on CAY allows academics that produce fewer, but more impactful papers, and work in smaller teams, to enter the ranking. However, unless one limits the comparison to a relatively short period of time, a similar set of-mainly older-academics is still likely to dominate the rankings over the years as their papers have had a longer time to gather citations. In comparison to the two other rankings, the hIa ranking is therefore likely to be most dynamic over time. Younger academics can more easily enter into this ranking if they perform well relative to their career length in terms of single-author equivalent impactful papers. It is also more dynamic in that currently ranked academics need to sustain their level of single-author equivalent impactful publications to remain listed; their hIa will decline over the years if no additional high impact papers are published.. Any ranking has its limitations, as does any metric on which the ranking in question is based. The nature of rankings means that even small differences in the metrics used can have an important impact on the position of individual academics (or departments or universities). This is clearly evident in the original Economics top-40 where the rank of individuals varied substantially depending on whether publications were multiplied by the journal impact factor, the raw article influence score or the percentile article influence score (variations that were applied in the methodology in recent years). Our own rankings made a conscious choice for author and age corrected metrics in order to provide more equitable comparisons between sub-disciplines with different authorship traditions and academics at different career stages.. However, rankings-especially rankings of individuals-are also vulnerable to minor variations in data input, even one additional publication or citation can make a difference in rank. This is especially true in the lower regions of any ranking; the variance in the top-20 for all three rankings discussed in this paper is substantially higher than the variance in the bottom top-20. In fact, the range of scores in the top-10 for any of the three rankings makes up 73-81 % of the total range of scores in the top-40, suggesting that beyond the top-10 differences are marginal and that it might be better to use bands of scores beyond the top-10. Beyond identification of the absolute top in a particular discipline, any differences in rank are due as much to specific ranking criteria, choice of metrics and the source of data. Rankings should therefore be considered for what they are: crude 7 Please note that this is a limitation that also applies to the ISI database. Although one can limit the year range for articles, it is not possible to do so for citations. Scientometrics (2015) 102:727-749 747 instruments to identify top performers, but should never be used as the only input for decision-making. (1) Academics listed in green bold font also feature on both new citation-based rankings, i.e. they are ranked on all three rankings. (2) Academics listed in orange bold font feature on the current Economics top-40 and on the PoP CAY ranking, i.e. they are ranked on two rankings. (3) Academics listed in red bold font feature on the current Economics top-40 and on the PoP hIa ranking, i.e. they are ranked on two rankings. (4) Academics listed in black regular font are unique to the current Economics top-40 and do not feature in either of the citation-based rankings.. Only 11 academics feature in all three rankings, whereas 24 academics feature in two of the three rankings. Forty academics feature on only one ranking, 18 only in the original Economics top-40, 10 only in the PoP CAY ranking and 13 only in the PoP hIa ranking.. The average age for academics in the Economics top-40 is 50 (range 39-67) and on average, they have 111 CAY in Google Scholar. Many academics in the original Economics top-40 have well under 100 CAY, the approximate cut-off score for the new PoP CAY top-40 discussed below. All but one of the academics in the original ranking are male and the vast majority (34) is Dutch, with four Flemish academics, one Greek and one Australian. Half of the academics (20) in the top-40, and no less than three quarters of the academics in the top-20 work in the Economics discipline. Management Science (8) and Marketing (6) are the second and third most frequently listed disciplines. Management (4) and Finance & Accounting (2) close the ranks. In terms of institutions, Erasmus tops the rank with 11 occurrences, followed by Tilburg (9). The VU (VU University Amsterdam) (7) and Groningen (6) follow third and fourth, with Wageningen (3) Maastricht (2) and Eindhoven (1) closing the ranks. UvA (University of Amsterdam), Nijmegen and Utrecht do not have any academics featuring in the top-40.. Nearly half of the academics in the current Economics top-40 drop out of the top-40 when ranked by the number of CAY for 2003 -2013 or the whole-ofcareer hIa (PoP hIa top-40). However, only two of the current top-10 (Goos and Oude Lansink) drop out in our new citation-based rankings and only five of the current top-20 do not feature in our new citation-based rankings. Out of the lower ranks (21-40), only seven make it to the ranking based on CAY or the whole-of-career hIa ranking; three of those seven are only just out of the top-20. Hence, in general academics listed at the top of the current Economics top-40 do indeed appear to be high performing on citation-based ranking criteria as well. However, there are some notable exceptions.. Peter Goos is ranked 7th in the Economics top-40, but, as can be seen in Tables 2 and 4 The high rank of Alfons Oude Lansink (#8) is a little surprising as the 28 articles he produced between 2008 and 2012 were mostly published in agricultural economics journals that do not have particularly high AIS. In any case, none of these articles, nor any of his other publications achieved high citation counts and as the articles were also normally co-authored with several co-authors the CAY metric is fairly low.. The two academics that experience the largest drop in ranking are Einmahl and van Dijke. Einmahl is ranked 28 in the Economics top-40, based on a ''mere'' 17 articles. However, all these articles were published in journals at the top end of the distribution of article influence scores such as Journal of Econometrics, Bernoulli, Annals of Statistics, and Journal of the American Statistical Association. Based on the raw AIS score he would even have been ranked 17th. However, his articles in these and other journals gathered very few citations; only eight of them attracted more than 10 Google Scholar citations overall.. Van Dijke has a similar profile in a very different field. He has published 25 articles in high AIS journals in Psychology, such as Journal of Applied Psychology, Organizational Behavior and Human Decision Processes, Journal of Experimental Social Psychology and Journal of Vocational Behaviour, but his publications in these and other journals have very modest citation levels and several co-authors. In van Dijke's case, his citation levels might be further reduced by the fact that he has published more in the last five years than between 2003 and 2008, reflective of a relatively junior academic who only started publishing in 2003.. A citation-based ranking: PoP CAY top-40 Table 2 provides details of our first new ranking, based not on the expected impact inferred from the journals in which the academic publishes, but on the actual article level citations that the academic attracts. The average age for academics in the top-40 of this ranking is 53 (range 37-73) and on average, they have 174 CAY in Google Scholar. Five of the academics are female and the vast majority (32) is Dutch, with three US academic, two Flemish academics, one Greek, one German and one Australian. Seventeen of the academics in the CAY top-40 work in Economics, with Management (13) the second most frequently listed disciplines. Finance & Accounting (4), Marketing (3) and Management Science (3) close the ranks. In terms of institutions, Erasmus tops the rank with 11 occurrences, followed by Tilburg (8) and the VU (VU University Amsterdam) (7). Maastricht (5), UvA (University of Amsterdam) (4), Groningen (3) and Eindhoven (2) close the ranks. Nijmegen, Utrecht and Wageningen do not have any academics featuring in the top-40.. In comparison with the old Economics top-40, average age and distribution of nationalities are similar. However, we do find a more diverse set of nationalities at the top. Three out of the top-4 are not Dutch, whereas first-ranked Richard Tol has had his main employment outside the Netherlands for many years. The proportion of female academics has also increased, although still low at 12.5 %. Reflecting its focus on citations over publications, the average number of CAY has increased by 57 %. In terms of disciplines, the proportion of academics in Management in particular has increased substantially, from 10 to 33 %, mainly to the detriment of Marketing and Management Science. In terms of institutions, Maastricht and Eindhoven have increased their representation, whereas the UvA (University of Amsterdam)-which had no academics in the old top-40-now has no less than 4 academics in the top-40. Groningen saw its listed academics halved, whereas Wageningen no longer has any academics listed.. The resulting PoP CAY 2003-2103 top-40 is substantially different from the publications-based ranking. The new top-10 only includes four of the same academics: Richard Tol, who ranks first in both rankings, Daan van Knippenberg, who ranks 4th on publications and 5th on citations, Peter Nijkamp, who ranks 6th on both publications and citations and Michael McAleer who ranks 2nd on publications and 10th on citations. Jan van Ours, ranked 11th in the original Economics top-40, is now ranked 7th. New in the top-5 are Thorsten Beck and Bronwyn Hall, who do not feature in the publications-based ranking at all; whereas Marno Verbeek and Roy Thurik are newcomers to the top-10. Eva Demerouti was listed in the old Economics top-40, but jumps from 17 to 3. We will discuss each of these academics in turn. Eva Demerouti illustrates a second aspect of our proof over promise approach: the more comprehensive journal coverage of Google Scholar. She already ranks high in a publication-based ranking and her publications in high impact journals are highly cited. However, what propels her to the top-3 in a citation-based ranking are three publications in journals that have no AIS and hence do not count at all for the original Economics top-40. Her publications in the International Journal of Managerial Psychology, Career Development International and the International Journal of Stress Management alone have gathered a total of more than 2,000 Google Scholar citations.. Bronwyn Hall illustrates a third aspect of our proof over promise approach: Google Scholar's coverage of non-journal publications. Eight of her twenty most highly cited publications are NBER working papers, book chapters or conference proceedings papers. In addition, many of these papers are single-authored. In addition, Hall's papers in ISI listed journals are all more highly cited than would be expected from their AIS.. Marno Verbeek features in the citation-based top-10 mainly based on a single-authored book ''A guide to modern econometrics'' which makes up 2/3 of his citation record. Some observers might argue that a top-10 ranking based largely on a publication that is not fully refereed is not justified. This is where an h-type indicator might provide a useful alternative viewpoint. Based on a whole-of-career hIa ranking (our second alternative ranking) Verbeek drops out of the top-40.. Finally, Roy Thurik illustrates a number of the above-listed aspects of our ''proof over promise'' approach. Like Bronwyn Hall, he has a large number of highly cited books or book chapters and working papers. However, he has also published about a dozen papers in the same journal: Small Business Economics, which does not have a particularly high AIS. However, he authored three of the journal's top-10 most highly cited papers.. Overall, our Google Scholar citation-based ranking clearly taps into a different aspect of scholarly performance than the ISI publication-based ranking. We would argue that a focus on impact over output and a more inclusive consideration of publication outlets produces a more relevant ranking of research excellence. The ranking is more relevant academically as it considers the actual impact of publications, i.e. the acceptance of the relevance of publications by academic peers. The ranking is also more relevant societally as it includes publication outlets beyond a narrow set of academic journals. In the next section we analyse the differences between the Economics top-40 and the PoP CAY top-40 in a little more detail. In our paper, we compare and contrast three different rankings: the Economics top-40 as published by ESB and two new citation-based rankings. The first alternative ranking provides a ranking based on citations to papers published in the last 11 years only (2003) (2004) (2005) (2006) (2007) (2008) (2009) (2010) (2011) (2012) (2013) . Citations are corrected for the number of authors and divided by the number of years to arrive at the citations per author per year (CAY). The second provides a ranking based on a whole-of-career individual annual h-index (hIa). This metric improves on the hindex by correcting citations for the number of authors as well as the length of an academic's publishing career. Further details on the sample, data source, data collection procedures and metrics used can be found below.

O O O O O O O O O O O O B O O O O O B O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O B I O O B O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A proof markup language for Semantic Web services Web services propose that they provide the means for remote interoperable access of components and software systems. However, successful inter-operation between components that do anything more than the simplest information retrieval is dependent upon those components having a shared understanding of the results that have passed between them. In this paper, we address the issue of understanding and trusting results generated by web services. We introduce a proof markup language (PML) that provides an interlingua for capturing the information agents need to understand results and to justify why they should believe the results. We also introduce our Inference Web infrastructure that uses PML as the foundation for providing explanations of web services to end users. We additionally show how PML is critical for and provides the foundation for hybrid reasoning. Our contributions in this paper focus on technological foundations for capturing formal representations of term meaning and justification descriptions thereby facilitating trust and reuse of answers from web agents. Web services are expected to make use of automated reasoners and retrieval engines to perform their services. In order for Semantic Web services to explain their results, they need to be able to generate justifications of their results in an exchangeable, combinable format. We refer to a sequential trace of information manipulations used to generate an answer as a proof of the answer. These proofs (possibly provided by multiple question answering systems) may be combined to form proofs of compound results. Proofs may be abstracted, summarized, or transformed into more abbreviated complex aspects or more understandable justifications for the answers. For the purposes of this paper, we will view an explanation as any transformation (abstraction or summaries included) of a proof that makes it more understandable to users.. In order to provide the foundation for proof manipulations -presentations, abstractions, summaries, transformations, and explanations -we introduce a proof markup language as a proof representation interlingua. This article introduces the Proof Markup Language (PML) that can be used as follows:. • To represent different kinds of proofs ranging from formal natural deduction derivations to proof traces typically produced by highly optimized theorem provers. These representations are aimed at services that leverage inference.. • To represent proofs resulting from services aimed at information retrieval or entity extraction. These proofs may provide information concerning sources and extractors use;. • To represent different kinds of explanations varying from summaries of assertions or knowledge sources used to derive an answer to a more elaborate abstraction of a derivation path;. • To support the exchange of proof information among automated reasoners in hybrid reasoning systems.. The rest of the article is organized as follows. Section 2 describes the Inference Web that is a Semantic Web infrastructure for explanations. Section 3 introduces the PML specification. Section 4 demonstrates practical uses of the PML format for explanations and hybrid reasoning. Section 5 presents related work. Section 6 concludes the article and describes future work.. • Specifications: Question answering components may generate answers and justifications for their answers using the PML format described in this article. PML provides a proof interlingua representation. Inference Web includes a specification of PML in DAML+OIL [2] and OWL [11] .. • Data: PML documents published on the Web become a portion of the Inference Web data used when browsing and summarization tools are presenting explanations, summaries, and other viewing options to users. Inference Web also processes the PML documents to identify when portions of them may be combined to form more complex conclusions. IW also uses rewrite rules to transform the proofs into more understandable explanations. The IW data includes the PML proofs and explanations along with a registry of information used for proof presentation. The registry, called IWBase, is a distributed repository of meta-data including sources, inference engines and inference rules. This information is used to provide follow-up information concerning the elements presented in proofs and explanations.. • Tools and Services: Inference Web includes a tool suite including a browser for displaying proofs and explanations, an explainer for abstracting PML proofs into PML explanations, a registrar for submitting and maintaining the evolving IWBase entries, a proof generation service for facilitating the creation of PML proofs by inference engines, and the IWBase registry for storing information used in proofs and explanations.. The IWBase [10] (formerly known as the IW Registry) is an interconnection of distributed repositories of meta-information relevant to proofs and explanations, including knowledge provenance information [13] . Every entry in these repositories is an instance of an IWBase concept as described in Section 3.3. For example, Knowledge Source is an IWBase concept that is useful for entries such as ontologies, knowledge bases, thesauri, etc. The knowledge source entry for an ontology describes stores of assertions about the ontology such as its original creator(s), date of creation, data of last update, version, URL (for browsing), description in English, etc. IWBase's provenance information is expanding on an as-needed basis driven by application demands.. Every entry has a URI and is stored both as a file written in OWL and as a set of tuples in a database. IWBase files are mainly used by PML proofs to annotate their content as described throughout this article.. PML classes are OWL classes (thus they are subclasses of owl:Class. They are used to build OWL documents representing both proofs and proof provenance information. Thus, PML concepts can be considered to be either proof level concepts or provenance level concepts. Primitive types mentioned in the article are from the XML schema specification 1 . In this article we have introduced a proof markup language -PML -that is used to support web services inter-operation and trust. PML has been integrated into the Inference Web infrastructure and is used as the interlingua that allows IW to present, combine, summarize, abstract, and explain answers generated by web services. In our discussion of hybrid reasoning and our integration efforts with JTP, we presented how PML supports hybrid reasoning and provided one implemented example of how the integration with a reasoner works. PML provides the representational foundation and Inference Web provides the tools and infrastructural support for enabling proof annotation, knowledge provenance presentation, and explanation browsing for agents and humans. This work provides the foundation from which web services may realize their promise of providing remote, interoperable access across components of distributed software systems. Automated reasoners have different ways to represent proofs [15] . Moreover, a reasoner can have multiple ways of representing proofs. For example, a reasoner can have an internal representation of proofs that has a number of features to handle optimizations and an external representation of proofs used to present a trace of how conclusions are derived. Proofs exchanged between reasoners are usually external representation of proofs. Indeed, many proof elements such as optimization properties can be useless for other reasoners because the optimizations are tied to the internals of a particular engine.. External representations of proofs have been developed for several reasons. For example, most automated reasoners are able to produce a simple trace of their proofs in order to debug the reasoners themselves. The need to check proofs is a more sophisticated reason to have a external representation of proof. For example, Watson [16] created a technique to represent and check proofs produce by Isabelle [12] .. Most external representations of proofs in use by more than one automated reasoning were developed within the context of hybrid reasoning systems and logical frameworks. But even in the context of hybrid reasoning systems, few are the efforts to create a general representation of proofs rather than a representation that is able to conciliate few reasoner-specific representations. PDS [1] is an example of a general representation of proofs that is used on hybrid reasoning systems such as MBase [7] .. PML and PDS share the ability of representing proofs at different levels of abstraction. For instance, the PML ability to generate explanations from proofs corresponds to the PDS notion of "third dimension" for is proof representation. The MBase use of PDS is also of particular interest for the PML since it demonstrated the need of a web artifact to represent proofs. Indeed, MBase use an XML version of PDS.. We have provided a few example web services that use PML proofs. The KSL Wine Agent 7 , the DAML Query Language Client 8 , and the OWL Query Language Client all use PML proofs in order to provide interoperable explanations of their answers. The IW Browser 9 is a web service that renders PML proofs and presents them in multiple formats for humans. All of these agent's use the Stanford JTP hybrid reasoner but we are extending SRI's SNARK theorem prover 10 to produce portable proofs and simultaneously integrating ISI's Prometheus 11 query planner with Inference web. We are also pursuing discussions with designers of other reasoning systems including W3C's CWM 12 and UT's KM 13 and SRI's SPARK.

O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O B B O B O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O B O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O B O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O B B O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Cyclic Orbit Codes Abstract-In network coding a constant dimension code consists of a set of k-dimensional subspaces of F n q . Orbit codes are constant dimension codes which are defined as orbits of a subgroup of the general linear group, acting on the set of all subspaces of F n q . If the acting group is cyclic, the corresponding orbit codes are called cyclic orbit codes. In this paper we give a classification of cyclic orbit codes and propose a decoding procedure for a particular subclass of cyclic orbit codes. Let F q be the finite field with q elements (where q is a prime power) and let n be natural number.. When U is a subspace of the vector space F n q we will use the notation U ≤ F n q . The dimension will be abbreviated by dim(U). The set of all subspaces of F n q of dimension k is called Grassmann variety or simply Grassmannian and is denoted by G q (k, n) := {U ≤ F n q | dim(U) = k}. The cardinality of this set is given by the q-Binomial coefficient, also called Gaussian number,. .. The union of all Grassmann varieties, i. e. the set of all subspaces of F n q , is called the projective space and denoted by P q (n) = n k=0 G q (k, n).

O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O


Presence, workload and performance effects of synthetic environment design factors There remains a limited understanding of factors in presence and its relation to performance. This research examined a range of synthetic environment (SE) design features (viewpoint, auditory cue type and visual background) suspected to influence presence, and evaluated differences in presence, workload and task performance caused by manipulations of the factors and task difficulty in a virtualreality-based basketball free-throw task. Thirty-two research participants were also required to perform secondary-monitoring tasks to assess attention allocation to the virtual and (surrounding) real environments, as an indicator of presence. Analysis of variance results demonstrated immersiveness (viewpoint) and auditory cue type to significantly influence the sense of subjective presence and perceptions of workload. Virtual task performance was significantly affected by task difficultly. This study also provided further evidence of significant positive relations between presence and workload, but no evidence of a correlation of objective presence and performance. These results have general applicability for the design of multimodal SE-based interfaces for real-world tasks, such as telerobot control. r Presence has been defined as the sense of being physically present within a computer-generated or remote environment, and has been identified as a design ideal for synthetic environments (SEs) (Draper et al., 1998) . When people refer to this subjective experience in the context of using an SE as an interface to remote actuators and sensors to control a distant machine or robot (i.e., perform teleoperation), the terminology ''telepresence'' is more commonly used. Both terms are used in this study; specifically, ''telepresence'' is used in relation to human control of teleoperators and ''presence'' is used in relation to human performance in virtual reality (VR) simulations. In general, an SE is used to provide computer-mediated human interaction with an environment that is physically separate from the user in order to allow human perceptual, cognitive and psycho-motor capabilities to be projected into normally inaccessible, hostile or simulated environments (Draper et al., 1999) . Since current technologies do not allow for the development of robots that have the information processing and motor control abilities of a human being, human operators remain involved in operation of robots at remote sites through computers or other advanced technologies for perceptual and cognitive tasks. This reality has caused a change in the direction of teleoperation research from the design of anthropomorphic robots (c. 1970 ) to a focus on human-computer (machine) interface design that provides high-fidelity displays, including rich visual, auditory and touch information on a remote site in order to facilitate operator perception and a sense of telepresence at the remote site (Riley, 2001 Abbreviations: ANOVA, analysis of variance; HMD, head-mounted display; ITQ, Immersive Tendencies Questionnaire; PQ, Presence Questionnaire; RW, real world; SD, signal detection; SE, synthetic environment; SSQ, Simulator Sickness Questionnaire; VR, virtual realitybased on the belief that the sense of telepresence will improve overall teleoperation (human-in-the-loop) task performance (Sheridan, 1992) . The findings presented above provide some basic insight into the potential presence implications of the design of VR or SEs for use in teleoperation tasks/systems; however, the results may be limited in terms of generalizability to specific real-world teleoperation scenarios. There is a need for elaborate investigations of the role of multimodal SE design factors in telepresence, performance and workload using tasks that are more representative of realistic SE applications. Contrary to our initial projection regarding the use of the basketball simulation in this study, it is possible that in a more complex VR application, involving rich sounds and visual information, presence experiences and fluctuations due to changes in system interface features may be even greater. As an example, in a teleoperated surgical environment implemented by Freysinger et al. (2002) , highly detailed audio and visual information was provided to users to promote a higher sense of telepresence and to promote operator performance. The absence of this information in the integrated SE could be detrimental to a surgeon's perception of involvement in an actual surgical operation and ultimately the accuracy of performance. Based on the present research, the use of auditory cues in SEs for telerobot control may serve to enhance user telepresence and performance. Furthermore, the viewpoint of a user may be critical to telepresence in the teleoperation environment.. One concern that can be raised, based on the findings presented here with respect to design for telepresence in complex teleoperation applications, is that the manner in which SE features combine or interact together (e.g., auditory cues, immersive viewpoint, visual fidelity) may not be additive in terms of a resulting affect on telepresence and synthetic task performance. Based on our results, the lack of interaction effects of SE factors on subjective presence measures and VR task performance suggests that design using multiple sensory channels could be limited to consideration of main effects of each channel on performance, presence, etc., as part of the interface development process. This needs to be further evaluated through observation of VR user experiences in higher fidelity simulations involving realistic SE applications.. Caution may be necessary in manipulating multiple sensory channels and features in SE-based interface design for complex systems to facilitate presence experiences, as there may be a mental workload ''side effect''. In the present study, the pattern of ANOVA results on, and correlations among, subjective mental demand ratings and telepresence in the virtual sporting task supported this contention. It is important to note, however, that mental workload was generally low across all experimental conditions (on a scale from 0% to 100%, all condition means were less than 60%). It is possible that users may be even more sensitive to SE feature manipulations and combinations of various viewpoint, auditory cue and visual background settings under more demanding teleoperation task circumstances.. Finally, the lack of correlation of presence experiences and performance in the present study should also be of concern is designing actual teleoperation systems. Since the virtual sporting we used primarily required psycho-motor behaviors of participants, it may not have been as sensitive of a paradigm for revealing a relation among the constructs, as compared to a teleoperation simulation requiring complex cognitive processes for manipulator path planning and tracking. These types of task performance requirements are critically dependent upon user achievement of situation awareness on the state of a robotic system and remote task environment that, in-turn, may be dependent upon users achieving self-fusion or telepresence with an SE-based interface. The results of this study provide evidence that the features of an SE, which include immersiveness (viewpoint) and auditory cue type, significantly influence the sense of subjective presence. These results are consistent with the findings of Slater et al. (1996) and Dinh et al. (1999) . However, our findings did not support those of Barfield et al. (1995) indicating that increased visual realism of an SE increases presence. It is possible that there is some minimum level of visual realism necessary for the sense of presence in VR; however, beyond that level, identifying differences among comparatively rich visual backgrounds in terms of presence may require more than rendering versus texturing and presentation of additional objects.. Unfortunately, in this study, the objective attentionbased measure of presence did not prove to be sensitive to the experimental manipulations, like subjective ratings of presence. This differs from previous findings, e.g., Riley and Kaber (2001) found a significant correlation between visual attention to VR displays versus elements of the surrounding reality and ratings of telepresence (PQ scores using Witmer and Singer's, 1994 measure) . This difference may be attributable to the unique design of the secondary tasks as part of the measure used in this study or differences in the experiment condition manipulations. However, as Draper et al. (1998) have observed, and as Singer (1994, 1998) demonstrated, since telepresence is a subjective experience that is significantly influenced by individual susceptibility, it may be that telepresence is most sensitively measured using subjective methods or ratings.. This study indicated level of task difficulty to be an important factor in performance in VR, but there was no corresponding significant effect of task difficulty on presence ratings. This finding also differs from the results of the studies by Riley and Kaber (2001) and Riley et al. (2004) . They explored multiple levels of difficulty in simulated teleoperation tasks, e.g., requiring participants to search for and detect land mines in an outdoor VR using a robotic rover. In Riley and Kaber's (2001) study, participants who were exposed to a high difficulty condition reported significantly lower (po0:05) telepresence (possibly as a result of task overload) than participants in low or moderate groups, which were not significantly different.. The results of the present study also indicated that viewpoint, sound and individual differences were significant factors in subjective workload. We suspected that the observed influence of the viewpoint condition on workload (as well as presence ratings) might also be attributable to the nature of the task, as dictated by the viewpoint (i.e., ease of observing the distance between the player and the goal). Mental workload was also significantly, positively correlated with subjective presence. This is consistent with the findings of the study by Draper and Blair (1996) , in which telepresence ratings were significantly correlated with composite workload scores during completion of a pipe-cutting task using a teleoperator. In our study, all VR factor settings that led to greater perceptions of presence also led to increased perception of mental demand. This is supported by the various statistical analyses. Other previous research (Riley and Kaber, 2001 ) has shown that telepresence may be significantly, negatively correlated with mental workload. However, some of the simulated teleoperation conditions in Riley and Kaber's (2001) study may have led to very high ratings of frustration, as part of the NASA-TLX, because of difficulty participants had in finding mines with the rover. This would have led to high overall workload scores and possible disengagement of participants from the task due to frustration. In general, all these results taken together indicate that some minimum level of workload may be necessary to develop a sense of presence; however, when workload is too high, it may degrade telepresence. Finally, in the present study, the mental workload ratings were significantly, negatively related to performance. It appeared that participants performed worse, although they worked hard (such as thinking and calculating the force to shoot a basket) in the simulation.. This study also revealed a significant correlation between ITQ and the subjective ratings of presence, but not among ITQ scores and the objective measure of presence, based on secondary-task performance. These results were not unexpected as the attention-based objective measure of presence did not appear to be a sensitive indicator of changes in presence due to the manipulations of factors anticipated to underlie presence experiences (which were revealed by the subjective ratings). Rather, the ratio of SD performance in secondary tasks in VR and reality appeared to be more of an indicator of the primary task load than attention allocation to displays toward achieving presence. The findings on the objective presence measure also indicated that Draper and Blair's (1996) simple PQ may be a quick and valid subjective assessment of presence relative to immersive tendencies identified by Witmer and Singer (1994) .. The subjective presence measure was not correlated with virtual task performance. This might have been expected based on the primary ANOVA results of the study demonstrating task difficulty to be important to performance, but sound and viewpoint to dictate presence ratings. This differs from the results of Zhang et al.'s (2005) recent study, in which they found the addition of 3-D auditory or visual feedback to improve virtual assembly task performance. Unfortunately, they did not measure presence in the virtual task. Although the shot distance in our task significantly influenced task performance, it did not appear to influence presence ratings. The study also did not reveal a correlation of objective presence and performance. These results differ from the findings of previous research showing significant correlations of presence and performance Riley and Kaber, 2001 ). The differences maybe attributed to the specific experimental manipulations or the nature of the task. In this study, we selected independent factors and settings that created the potential for presence and performance to function in different ways in order to, in part, verify presence as a unique construct. It is also possible that since the basketball simulation primarily required psycho-motor performance (distance estimation, shooting the ball) and required more participant interaction skills, presence may not have been as critical to performance as in a telerobot control scenario, like that explored by , requiring cognitive functions including manipulator path planning for which immersion of one self in an SE interface may be key to performance.. There were no significant interaction effects among all factors (visual background, sound, viewpoint and task difficulty) manipulated in this study in terms of performance, presence and workload. Although counter to expectation, and our hypothesis that a cue conflict might occur as a result of auditory cue and visual background manipulations (i.e., a cheering crown with no visual of an audience in the virtual stadium), this result is consistent with the findings by Dinh et al. (1999) . They investigated the effects of tactile, olfactory, audio and visual sensory cues on a participant's sense of presence in a virtual environment that consisted of a corporate office suite including a reception area, hallway, bathroom, small office, copier room, large office and balcony. Their analysis revealed no significant interaction effects on subjective measure of presence. In the present study, information from the different sensory channels (e.g., auditory and visual) did not interact to improve or decrease overall performance and presence in the VR, nor did the combination of cues influence the subjective perception of workload. These results do not support the intuitive hypothesis on cross-modal interaction in VR by Biocca et al. (2001) that multiple sensory cueing of events in an SE will lead to increased presence experiences. However, the current study did not explore haptic cues, like Biocca et al. (2001) . It is possible that there is some interaction between the visual and haptic modalities that may be critical to presence in VR tasks for which motor control is a significant part. It may be more likely, as Stanney (2003) contends, that the interaction (or general importance) of multimodal cues in VR design depends on the demands of the task and designers may need to identify those cues that best support performance.

B B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O B I O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O