EC3: Elastic Cloud Computing Cluster This paper introduces Elastic Cloud Computing Cluster (EC3), a tool that creates elastic virtual clusters on top of Infrastructure as a Service (IaaS) Clouds. The clusters are self-managed entities that scale out to a larger number of nodes on demand, up to a maximum size specified by the user. Whenever idle resources are detected, the clusters automatically scale in, according to some predefined policies, in order to cut down the costs in the case of using a public Cloud provider. This creates the illusion of a real cluster without requiring an investment beyond the actual usage. Two different case studies are presented to assess the effectiveness of an elastic virtual cluster. The results show that the usage of self-managed elastic clusters represents an important economic saving when compared both to physical clusters and to static virtual clusters deployed on an IaaS Cloud, with a reduced penalty in the elasticity management. The usage of clusters of PCs as a computing facility is currently widespread in the scientific community. In the last years, the success of this computing platform, either for High Performance Computing (HPC) or for High Throughput Computing (HTC) has been unparalleled. However, one of the main drawbacks of these computing platforms is the relatively large upfront investment together with the maintenance cost. For small and medium-sized research groups or organizations the purchase of such an equipment might represent an important cost.. Traditionally, virtualization was not considered as a viable option for HPC, mainly due to the overhead costs in I/O and network devices. However, the major improvements in hypervisor technologies have paved the way for Cloud computing to rise as a paradigm where resources (in the shape of virtual machines (VM), network, storage capacity, etc.) can be dynamically provisioned and released on a pay-as-you-go basis [1] . This is the case of public Infrastructure as a Service (IaaS) Cloud providers such as Amazon Elastic Compute Cloud (EC2) or Rackspace.. In a previous work [2] we concluded that, in some cases, it is interesting to deploy a virtual cluster instead of a physical one. A virtual cluster in an IaaS provider is able to get a competitive performance per price rate, but also gets important benefits from the Cloud provider such as reducing the administration costs (both personnel costs and maintenance of equipments), avoiding hiring or buying the physical building to host the infrastructure, avoiding the upfront investments in hardware, cooling systems, etc. Therefore, deploying a cluster in the Cloud can also inherit these advantages.. However, clusters are generally not used at 100% of its capacity during their lifetime [3] . The total lifetime of a cluster can be divided into two parts: the time while the cluster is calculating (T c ) and the time when the system is idle (T i ). In a Cloud environment it would be adequate to stop the VMs while they are not being used and pay only for T c . The idea is similar to what is currently done with energy saving techniques in a datacenter, where physical nodes are dynamically powered on and off in order to reduce energy consumption while maintaining the required level of service. However, whereas for Green computing the aim is to save energy, for Cloud computing the main aim is to save money (in the case of public Cloud providers). In this case, the Green computing techniques seem to be suitable for creating elastic clusters in an IaaS cloud deployment.. This article proposes the combination of Green computing, Cloud Computing and HPC techniques to create Elastic Cloud Computing Cluster (EC3), a tool that creates elastic virtual clusters on top of IaaS Clouds. EC3 creates elastic cluster-like infrastructures that automatically scale out to a larger number of nodes on demand up to a maximum size specified by the user. Whenever idle resources are detected, the cluster dynamically and automatically scales in, according to some predefined policies, in order to cut down the costs in the case of using a public Cloud provider. This creates the illusion of a real cluster without requiring an investment beyond the actual usage. Therefore, this approach aims at delivering cost-effective elastic Cluster as a Service on top of an IaaS Cloud.. The remainder of the paper is structured as follows. First, section 2 describes the related work and the main contributions of EC3 to the state-of-the-art. Later, section 3 details the architecture and implementation details of EC3. Then, section 4 describes two case studies to demonstrate the functionality of EC3 both in the case of a stable cluster and in the case of an ad-hoc cluster for a specific application. Finally, section 5 concludes the paper and points to future work. This article has introduced Elastic Cloud Computing Cluster (EC3), a tool to create HPC clusters on top of Cloud infrastructures that, using Green computing concepts, creates a self-managed system that dynamically scales to adapt to the workload of users.. The tool builds on top of CLUES, an energy manager system for cluster-like infrastructures, enabling the cluster to gain elastic capabilities on a Cloud deployment. For that, a Cloud connector has been developed to manage the interaction with the infrastructure manager in charge of starting and terminating the virtual machines that correspond to the internal nodes. Moreover the CLUES scheduler has been modified to adapt its policies to be able to manage elasticity to create virtual elastic clusters.. EC3 enables end users to deploy elastic clusters of a given maximum size in a matter of minutes with just a command line tool. Besides, the policies to start and shutdown nodes have been contributed back in order to enhance CLUES enabling more possibilities in order to consider the tradeoff between minimizing the impact in the waiting time for the user (which implies having a larger number of nodes started) and the reduction of the Cloud costs (which implies having a reduced number of running virtual machines).. Future works include several research lines. In the case of commercial Clouds it is crucial to incorporate cost-aware schedulers that consider not only the number of nodes but also the cost of the instances in order to better manage the budget without surpassing the specified limits (daily, monthly, etc.) specified by the user. Concerning the costs, it is important to consider deploying multi-core virtual machines that can share the execution of several jobs. This would enable reduced prices per execution unit. Finally, it is important to consider the case for heterogeneous cluster computing, where each node could exhibit different capabilities (more storage space, different CPU type, support for GPUs, etc.). This work encompasses the dynamic deployment of virtual elastic clusters on the Cloud. Regarding the creation of clusters in the Cloud, some works such as [4] , [5] and [6] have analyzed architectures, algorithms and frameworks to deploy HTC clusters over private, public and hybrid Cloud infrastructures. In [4] the authors analyze the performance of virtual clusters deployed on top of hybrid Clouds obtaining good results that demonstrate the feasibility of these kind of deployments. The work by Wei et al [5] is focused on algorithms to deploy VMs over a set of physical resources in the most efficient way trying to obtain the best performance on the virtual cluster. In [6] , the authors try obtain the best resource allocation solution for a set of virtual clusters from different users with different job features. However, none of them deals with the elastic adaptation of the cluster size to the workload submitted by their users. 2. In [7, 8] , the Nimbus toolkit is employed to implement and evaluate an elastic site manager, which dynamically extends existing physical clusters based on Torque with computational resources provisioned from Amazon EC2 according to different policies. A similar approach is employed in [9] , where the benefits of using Cloud computing to augment the computing capacity of a local infrastructure are investigated, but no details about the underlying technologies are given.. The standard distribution of Hadoop [10] includes a utility to create a virtual cluster in the Amazon EC2 infrastructure, managed by the Hadoop middleware. The utility powers on the master virtual machine, using a pre-defined Amazon Machine Image (AMI) and creates the computing nodes using another AMI, performing the required network configuration. It is possible to add new computing nodes to the running cluster. However there is no additional support to remove the nodes from the Hadoop cluster. Finally, when the cluster is no longer needed, it is immediately terminated in order to free the allocated resources. ViteraaS [11] allows creating virtual clusters on hybrid Clouds. This software enables the submission of jobs that need to be run in a cluster to the system. The middleware creates a cluster to fit the job and manages the execution of the job. The main problem is that ViteraaS does not allow the user to remotely access the cluster. Instead, ViteraaS is devoted to execute jobs as done in classic Grid approaches without providing access to a cluster. StarCluster [12] enables the creation of a Sun Grid Engine based cluster in the Amazon EC2 infrastructure, following a predefined configuration of applications (Sun Grid Engine, OpenMPI, NFS, etc.). It includes a plugin system that enables the user to add new elements to be installed on the cluster nodes, but the VMs are based on a pre-defined Amazon AMI. As it happens with Hadoop, the creation of the cluster is made from a User Interface (UI) that connects to EC2 and starts the instances, prepares the network, configures the middleware, etc. Along with StarCluster a plugin called Elastic Load Balancer [13] has been developed that is able to grow and shrink the cluster according to the length of the cluster's job queue. The caveat of this plugin is that it requires the StarCluster UI to be connected to the Cloud infrastructure, in order to create and destroy the VMs. Thus, any failure on the UI results in the loss of control of the elasticity capabilities of the cluster.. In the case of commercial solutions we can find different approaches to create virtual clusters. One example is IBM Platform Dynamic Cluster [14] that aims at partitioning the owned resources to deliver each user a custom cluster with specific features by using virtual machines. This is different from common clusters where any application or library is installed on every node and is available for every user, and it is sometimes hard for administrators to install different applications or libraries at the same time. The virtual machines ease this situation and enable user activity isolation. The drawback in this case is that this product is oriented to manage on premise infrastructures and cannot be connected to commercial Cloud providers. Another example of commercial solutions is CycleCloud [15] that is a service provided by CycleComputing that deploys virtual clusters, but it is restricted to Amazon EC2. This service provides the user with a virtual cluster based on SGE, Torque or Condor where a subset of popular scientific applications, offered by CycleCloud, are installed. The user is able to manage the virtual nodes using a web interface, and it is possible to configure the cluster so that it is automatically sized based upon pending jobs.. The summary is that several solutions have appeared for different environments, but there is no general framework that enables the creation and management of elastic clus-3 ters in general IaaS deployments. Moreover (i) most of them provide a virtual cluster that comprises a fixed number of nodes; (ii) each of the solutions is oriented to a specific LRMS since they take advantage of the individual features of such implementation; and (iii) most of them are oriented to Amazon EC2 and do not consider other public IaaS deployments, or even on-premise Cloud deployments (e.g. based on OpenNebula, OpenStack, Eucalyptus, etc.).. The main advantage of our approach consists in the development of a generic framework to create and manage elastic clusters that dynamically adapt to the current workload. The deployed clusters turn into self-managed entities where elasticity is handled from the cluster itself, without requiring external entities to monitor and act upon the cluster. In this way, a cluster on a Cloud dynamically shrinks and grows according to predefined policies without human intervention. In addition, the tool seamlessly harnesses resources from public and on premise Clouds.

O O B I O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Secure web services using two-way authentication and three-party key establishment for service delivery With the advance of web technologies, a large quantity of transactions have been processed through web services. Service Provider needs encryption via public communication channel in order that web services can be delivered to Service Requester. Such encryptions can be realized using secure session keys. Traditional approaches which can enable such transactions are based on peer-to-peer architecture or hierarchical group architecture. The former method resides on two-party communications while the latter resides on hierarchical group communications. In this paper, we will use three-party key establishment to enable secure communications for Service Requester and Service Provider. The proposed protocol supports Service Requester, Service Broker, and Service Provider with a shared secret key established among them. Compared with peer-to-peer architecture and hierarchical group architecture, our method aims at reducing communication and computation overheads. Resource delivery can be any activity through networks involving service request and provision. It may be for individuals, companies or government agencies. Because of the various possible attacks through heterogeneous networks [1, 5, 7, 13, 45, 28, 34, 35] , resource delivery requires some security mechanisms to protect data privacy associated with the requested resource. Service requester and service provider are distributed over different networks and domains. Therefore mechanisms for preventing eavesdropping of data from unauthorized parties have to be considered in any resource delivery which may be sensitive [1 -7, 9, 19, 48, 22, 47, 39, 40] .. Consider the following scenario for an e-government: Suppose the Taxation Office is a service requester. The Taxation Office (TO) may request an individual visa and employment history. The service provider can be the Department of Education and Employment(DEEW). The Department of Immigration and Citizenship (DIC) can act as a broker. Thus, DEEW can provide TO with the requested resource via DIC. Since DIC is a trusted party, the communications between TO and DEEW can be delivered via DIC. Another example is: You may use your mobile phone to direct your home computer to transfer your meeting schedule to your office printer and print out this schedule for you. Your mobile phone will act as a service requester and your office printer is a service provider while your home computer is a broker in this scenario.. In the above scenarios, the three parties in each situation can share one same session key for their resource delivery service. In this paper, we will develop a secure service protocol for this kind of resource delivery. Our method is based on non-commutative monids and one-way hash function. We will propose a two-way authenticated and three-party key establishment protocol which can address the above two scenarios.. A key establishment protocol is used to derive a shared secret by two or more parties as a function of information contributed by, or associated with, each of these, but no single party can predetermine the resulting value. The security of most key establishment protocols were based on large integer factoring infeasibility or discrete logarithm infeasibility [41] . Several key establishment protocols based on group theory have been proposed [8, 49, 12, 30] . However, all these protocols are for two parties to establish a secret key. These schemes cannot be transferred to the three-party scenario in their present forms. Therefore, it is interesting to propose three-party key establishment based on non-commutative monoids. In addition, the proposed key establishment should be immune from the existing attacks on algebraic method based cryptographic primitives.. It is true that the WS-world was not initially expected to serve in lowcomputation devices or similar scenarios that may have particular restrictions in their overhead, but in servers providing these WS-enabled services. However, we notice that the increasing popularity of the mobile devices such as laptop, PDA, and tablet type devices allow new opportunities to develop solutions beyond mobile email including other activities of web services. Therefore, it also opens a new window for delivering the mobile infrastructure which can support mobile web services. Apparently, low communication and computation overheads are positive for mobile web services. That is a sound motivation behind our proposal for the presented two-way authentication and three-party key establishment. There are of course many other motivations. One of them could be reducing time associated with secure web services arising from high overheads.. Some basic security requirements for resource delivery where three parties (service requester, broker and service provider) are involved are identified as follows:. • Requirement 1. Authentication: Before the resource delivery is executed, the participated parties need to be authenticated. That is, Service requester needs to be authenticated to Broker and Service provider. Broker needs to be authenticated to Service requester and provider. Service provider needs to authenticated to Broker and Service requester. All the authentications are required to be two-way authentication [5, 7, 11, 16, 17, 31, 40, 42 ] .. • Requirement 2. Data Privacy: In the whole course of resource delivery, data privacy should be protected. It has two requirements: on the one hand, data associated with the resource should not modified while being delivered; on the other hand, data associated with the resource should be kept unknown to any unauthorized party [18, 21, 46, 32, 42] .. Requirement 1 can be attained through password-based authentication which is adopted in our proposed two-way authentication in section 4. Requirement 2 can be attained through session key establishment and secure resource delivery using the established session key.. We need to clarify that other security requirements such as confidentiality, authorization, integrity and non-repudiation (see [41] for the details of these definitions) are also needed for full and secure deployment of the above two scenarios. However, our proposal only focuses on data authentication and privacy.. Other techniques such as encryption and digital signature can solve confidentiality and authorization or non-repudiation requirements [41] . Access control techniques and policies can address the authorization requirement [41] . Therefore, we did not address them in the proposed protocol.. The use of web services can pose serious risk if security is not properly ad- dressed from the beginning. The purpose of our proposal is not to examine current web service security standards but to facilitate secure web service transactions within a service oriented architecture by using our three-party schema which does not rely on a static and pre-built secure communication channel. Web service networks are dynamic and grow organically, which increases the cost of management and administration. Because of the dynamic nature, it is not realistic to rely on a static and pre-built secure communication channel for some concrete applications such as those two scenarios mentioned above. Our proposal can be combined with some of current web service security standards such as Security Assertion Markup Language (SAML) [5], XML Encryption [3] , and XML Signature [2] to enable authorization information, end-to-end encryption, and message integrity. In this paper we proposed a secure web service scheme for service delivery. This protocol used two-way authentication and three-party key establishment to realize the security requirements for service delivery in network environments. Our method can help to reduce communication and computation overheads. This is because in our model there is only one shared key was established for Service requester, Broker and Service provider. After two-way authentication, they can use this key for the service delivery using symmetric encryption. Symmetric encryption is more efficient than asymmetric encryption with regard to computation cost. We have discussed the shared keys' security, the security against some existing attacks on algebraic method based key establishment and the spoofing resistance. In this section, we will review some existing schemes which provide secure web services for service discovery and low-cost devices.. If there are three parties in the service delivery process, then peer-to-peer based model leads to two different shared keys which would be established. See the example as in Figure 1 .. Helander and Xiong proposed a secure web service framework based on peerto-peer model [23, 25] . That framework provided two-way authentication and peer-to-peer key exchange. Their method were based on RSA and AES [41] . The two-way authentication and peer-to-peer key exchange were done with RSA. The transmission and communications were encrypted with an AES peer-to-peer key. RSA-based method is well-known energy-consumed due to its large number involvement (e.g. 2048 bit-length number may be needed for maintaining some security level) [41] .. Campo et al.'s security model is based on hierarchical group communication [48] . The authors developed a multiprotocol service discovery platform. They studied mechanisms to enhance availability of services in the network. They did not present a concrete scheme for their framework.. The protocol in [51] was the service discovery chosen by the Bluetooth SIG to search for services in Bluetooth networks. Clients connect to all the prior discovered devices in the proximity and check if they are offering the desired service. This protocol uses a very simple scheme where security relies on a secure pairing between devices and scalability is limited to a low number of devices. It only works for Bluetooth networks and there is no way to run it on any other network technology. These features are not enough in terms of security and scalability.. The protocol in [36] supported multicast mechanism, by which the client multicasts a template of the service and any server receiving the message compares the pattern with its own services. If any service matches, the server answers to the client with the address of the service. The drawback of this protocol is: every client needs to maintain the public key of every server in the network. Another issue is sending the address in clear makes IP spoofing possible [41, 42] .. Authentication and key establishment can be realized using either the symmetric key method or the public key method [41] . The first symmetric key protocol involving online trusted third parties was proposed by Needham and Schroeder in [29] . This protocol was shown to be insecure by Denning and Sacco in [37] . The insecurity is because of one party being not able to ensure messages are fresh. Denning and Sacco suggested fixing the freshness flaw in [29] by the use of timestamps.. Otway and Rees in [43] proposed another symmetric key based protocol which involves an online server. This protocol was compromised by Burrows et al in [14] by fooling one party to believe a public message is a new established key.. Woo and Lam in [27] proposed another two-way authentication and key establishment protocol. This protocol was compromised by Clark et al. in [24] using a parallel session attack. The parallel session attack can enable party A to fool party B to accept a previously established key as a new one. This attack is not a particularly strong attack but indicates that the protocol does not provide the authentication property. Lowe in [33] developed a more serious attack using message component symmetry.. Needham and Schroeder in [29] proposed a public key based protocol which involves an online trusted third party. The public key based method needs the deployment of Public Key Infrastructure (PKI). Therefore, it is expensive from a system cost point of view. This protocol was broken by Lowe in [33] . Neuman and Stubblebine presented another protocol in [20] which was broken by Hwang et al. in [10] .. Compared to all the above protocols, the ISO/IEC protocol [50] is an efficient one which is based on the symmetric key method without an online trusted third party. Our proposal will use the method of symmetric key protocol without an online trusted third party.. In order to address the above issues, we will propose a new web service protocol which uses two-way authentication and three-party key establishment for resource delivery. Our protocol is targeting at three-party resource delivery scenario, i.e. there are three participants, Service requester, Broker, and Service provider. They only need to establish one shared key for fulfilling the resource delivery securely. The figure 2 gives an overview for this scenario as above:

O O O O O O O O B I O B I O O O O O O O O O O B O O O O O O B I O B O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O B I O O O O B I O O O B I O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O B I O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I B O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O B O O O O O O B O O O O O O O O O O O B O O O O O O O O O O B I O O B I O O O O O O O B I O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O B I O O B O B O O O O O B O O O O O O O O O O O B O B O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O


Sensor selection for energy-efficient ambulatory medical monitoring Epilepsy affects over three million Americans of all ages. Despite recent advances, more than 20% of individuals with epilepsy never achieve adequate control of their seizures. The use of a small, portable, non-invasive seizure monitor could benefit these individuals tremendously. However, in order for such a device to be suitable for long-term wear, it must be both comfortable and lightweight.Typical state-of-the-art non-invasive seizure onset detection algorithms require 21 scalp electrodes to be placed on the head. These electrodes are used to generate 18 data streams, called channels. The large number of electrodes is inconvenient for the patient and processing 18 channels can consume a considerable amount of energy, a problem for a battery-powered device.In this paper, we describe an automated way to construct detectors that use fewer channels, and thus fewer electrodes. Starting from an existing technique for constructing 18 channel patientspecific detectors, we use machine learning to automatically construct reduced channel detectors. We evaluate our algorithm on data from 16 patients used in an earlier study. On average, our algorithm reduced the number of channels from 18 to 4.6 while decreasing the mean fraction of seizure onsets detected from 99% to 97%. For 12 out of the 16 patients, there was no degradation in the detection rate. While the average detection latency increased from 7.8 s to 11.2 s, the average rate of false alarms per hour decreased from 0.35 to 0.19.We also describe a prototype implementation of a single channel EEG monitoring device built using off-the-shelf components, and use this implementation to derive an energy consumption model. Using fewer channels reduced the average energy consumption by 69%, which amounts to a 3.3× increase in battery lifetime.Finally, we show how additional energy savings can be realized by using a low-power screening detector to rule out segments of data that are obviously not seizures. Though this technique does not reduce the number of electrodes needed, it does reduce the energy consumption by an additional 16%. Epilepsy is a relatively common chronic neurological disorder characterized by recurrent unprovoked seizures. Epilepsy affects over three million Americans of all ages, at an estimated annual cost of $12.5 billion in direct and indirect costs. Despite recent advances in the management of epilepsy, more than 20% of individuals with epilepsy never achieve adequate control of their seizures [24] . A portable, non-invasive seizure monitor could benefit these individuals tremendously. For instance, it could be used to alert patients or caregivers of seizure onset before symptoms of the seizure cause injury [31] . In addition, an ambulatory seizure onset monitor could be used to automatically initiate delivery of a therapy to reduce both seizure intensity and duration [33] . Both applications require timely detection of seizure onset.. Seizure onset is defined as the point in time where a patient's EEG transitions from a non-ictal state to an ictal state. A seizure is the period in time following this event and typically lasts from one to two minutes. Note that the baseline EEG signals of a person with epilepsy are considerably different from the EEG signals of a person who does not have epilepsy [23] . Even among patients with epilepsy, the amplitude, spectral content, and spatial distribution of EEG signals can vary during both seizure and non-seizure periods.. Two key issues in designing an ambulatory seizure onset monitor are patient comfort and convenience. Several factors have an impact on these issues including:. (a) Number of electrodes. State-of-the-art seizure onset detection algorithms require 21 scalp electrodes to be placed on a patient's head to acquire EEG signals. The electrodes are used to generate 18 data streams called channels. Reducing the number of electrodes will help to improve the long-term wearability of a seizure monitor.. (b) Number of wires. Using wires to connect sensing components to computation components is cumbersome for a patient to wear. The use of a small number of wireless integrated, small and lightweight devices is likely to be more comfortable.. (c) Electrode interface. Current electrodes used for scalp EEG monitoring require the use of electroconductive gel to provide a low impedance interface between the electrode and the surface of the patient's scalp. Wearing gel electrodes for a prolonged period of time is uncomfortable. One potential way to mitigate this problem is to use dry electrodes [18] . However, since dry electrodes are larger in size (≈ 2.5 cm in diameter) than conventional electrodes (≈ 0.5 cm in diameter), they may be uncomfortable for weight or size reasons.. (d) Device weight. Since the device is designed to be worn on the head, the weight of the overall device should be minimized as much as possible.. In this paper, we focus on the problem of channel reduction. We present a machine learning based algorithm to construct reduced channel seizure onset detectors. Starting from an existing algorithm for constructing 18 channel patient-specific detectors, we use our technique to build reduced channel detectors. While using fewer channels will improve wearability and reduce energy consumption and device weight, the potential downside is that detection performance may degrade.. To select channels, we use an instance of the wrapper approach [13] , a feature selection technique. Because EEG signals can vary considerably among patients with epilepsy, a patientspecific seizure onset detector trained using data from a single patient tends to have better specificity and sensitivity than a nonpatient specific detector [27] . Thus, for selecting a channel subset for seizure onset detection, we elected to use our technique on each patient separately.. Using our approach, we reduced the average number of channels needed to detect seizure onset from 18 to 4.6. For 12 out of the 16 patients, the subset method detected all of the seizures detected by the original method. When we ran these detectors on recorded EEG data, we observed only a small degradation in performance as compared to the original 18-channel detector. The mean fraction of seizures detected decreased slightly from 99% to 97%. While the average detection latency increased slightly from 7.8 s to 11.2 s, the average number of false events per hour decreased from 0.35 to 0.19.. We also describe a prototype implementation of a single channel EEG monitoring device built using off-the-shelf components. Using this implementation, we derived a model that was used to evaluate the energy consumed by our detectors. By using fewer channels, our algorithm reduced the energy consumption by an average of 69%, which amounts to an increase in battery lifetime of 3.3×.. Finally, we show how to further reduce energy consumption by adding a screening detector to the reduced channel detector. By reusing the wrapper approach, we constructed a patient-specific screening detector for each patient. The combination of the patientspecific screening detector with the original reduced channel detector consumes 16% less energy while decreasing the percentage of seizures detected by only 1%. Medical researchers have developed good algorithms for extracting clinically useful information from physiological sensors. In order to deploy these algorithms for wearable medical monitoring, e.g., for seizure onset detection, the sensors and monitoring devices must be comfortable enough for prolonged wear. To make a wearable seizure onset monitor more comfortable, both the device weight and number of scalp electrodes need to be minimized.. In this paper, we described a machine learning-based technique to automatically construct a reduced channel detector. Starting from an existing technique for constructing 18 channel patientspecific detectors, we showed how to use machine learning to automatically construct reduced channel detectors. Reducing the number of channels improves wearability since fewer electrodes are less intrusive and are easier to mount. Moreover, reducing the number of channels also reduces the energy used by the monitoring device since less data will need to be processed.. Using data from 16 different patients, we automatically constructed patient-specific reduced channel seizure detection algorithms and compared their detection performances and energy consumptions to those of the original 18 channel detector. On average, only 4.6 channels were needed. The detectors were able to detect 97% of all seizures, a degradation of only 2%. For three quarters of the patients, there was no degradation in the detection rate. While there was an increase in the average detection delay from 7.8 s to 11.2 s, the average rate of false alarms per hour decreased from 0.35 to 0.19.. We also described a prototype implementation of a single channel EEG monitoring device built using off-the-shelf components. Using this implementation, we derived an energy consumption model which was used to evaluate the energy consumed by our reduced channel detectors. On average, using fewer channels allowed our system to use 69% less energy than the 18 channel detector. This translates to an increase in battery lifetime of 3.3×. Finally, we showed how to achieve additional energy savings by using a screening detector to rule out segments of non-seizure data at low-cost. While screening does not reduce the number of channels that need to be worn, it can further reduce energy consumption by 16% while still detecting 96% of the seizures.. Finally, it is important to note that, while the focus of this paper is on selecting channels for seizure onset detection, the technique of selecting a subset of channels can be easily applied to other sensorbased domains where data from multiple sensors is used to perform detection. The effectiveness of the algorithm relies on the objective function chosen. Using an appropriate objective function to select a subset of sensors can be beneficial if resources such as energy or bandwidth are limited. One concern with the results we have presented is that the seizures and spatial distribution of the seizures for a patient will change over time. Thus, the channels selected based on previously collected data will be irrelevant as time progresses. Unfortunately, published information about how seizures change over years or decades is not available. Clinicians generally believe that adults with epilepsy tend to have stable EEGs. In our own clinical studies, we have successfully used a patient-specific detector trained using data collected over a year earlier to detect a seizure in real-time.. Another concern is that using scalp electrodes is simply too cumbersome for long-term monitoring use. Unfortunately, other methods of seizure onset detection that do not use EEG electrodes are more error-prone. For example, one could try to use electromyography (EMG), the electrical signals generated by muscle movement, to detect a seizure. There are several problems with such an approach. First, not all patients experience strong muscle movement during seizures. Second, deciding where to place the EMG electrodes will be patient-specific since patients experience seizures in different ways. Finally, muscle movements that indicate clinical onset of seizure often lag the initial electrographic seizure onset. Thus, even if it were possible to detect a seizure as a result of a change in the EMG signal, the detection latency would likely be larger. Reducing the power consumption of mobile devices and systems has been well studied by many researchers [14, 8, 16] . For the most part, a reduction in energy consumption is achieved through some form of duty cycling. Deciding when and how frequently to turn a device off often depends on the application. Typically, the decision to turn devices off is based either on user input or on other environmental changes. In many previous systems, power savings is achieved by trading off the performance or fidelity of certain applications (e.g., a low-resolution image is displayed or a task takes longer to complete). In this paper, we use software techniques to select channels in order to reduce energy consumption of seizure onset detection. However, unlike previous work, for many patients, we are able to achieve energy savings with only a slight degradation in the detection performance.. A few researchers have explored the use of feature selection to reduce the complexity or energy consumption of a detector or classifier. In [30] , the authors use a genetic algorithm to construct a decision tree where the fitness function takes into account the cost of performing a test and cost of making classification errors. The work is evaluated using medical data sets. Energy consumption is not considered since the medical tests are not intended for real-time use. Figure 19 : The average number of channels used in the screener was 1.17. For some patients, a screener was not needed since the subset algorithm used only a single channel. For other patients, the screeners tended to degrade detection accuracy on the training data. Thus, we did not use a screener for those patients.. More recently, Benbasat et al. take into account the energy cost of sampling various sensors for performing wearable gait monitoring [3] . They describe a framework for automatically generating power-efficient decision tree classifiers for wearable gait monitoring nodes. Similar to AdaBoost, decision tree learning algorithms perform embedded feature selection. By modifying the learning algorithm to take into account energy cost, a tree that is "energyaware" can be built. Once a decision tree is constructed, the system dynamically consults only the sensors that are necessary to determine the system state. However, because sensors are activated sequentially over time, the detection latency of such systems will be greater than those that examine all the data simultaneously. We have not yet tested the use of a decision tree algorithm for our application. However, decision tree classifiers tend to have lower specificity and sensitivity than SVM-based classifiers. Nevertheless, it may be worthwhile to compare these methods to ours.. The authors of [20] also explore ways to scale the data or feature set used by an ECG-based application for use on a mobile device. The authors show how to use their methods to trade off classification accuracy for bandwidth. In particular, they use a feature selection method based on mutual information to "scale" the data set such that computation is appropriate for a mobile device. Other researchers [10, 15] have applied feature selection techniques such as Recursive Feature Elimination for reducing channels in EEG-based applications.. The idea of using a lower energy, but lower fidelity device most of the time is similar to our concept of screening. In [25] , the authors use a low-power radio to receive incoming wake-up messages. This allows the rest of the device to remain off when the device is not being used and results in dramatic energy savings. The use of multiple radios to reduce energy consumption is further explored by various researchers [21, 1] who use Bluetooth or cell phone technologies to implement the wake-up mechanism. The wake-up mechanism is similar to screening in that a higher-power device is activated only when an incoming message is detected on the low-power channel.. Similar to wake-on-wireless systems, a number of researchers have built sensor-based hardware platforms designed to detect rare and infrequent events [7, 17] . In these systems, the CPU stays in a very low-power mode and is activated only when a sensor detects some signal energy.. Previously, we investigated the use of a screener detector to reduce energy consumption for the seizure detection problem [26] . In this paper, we use feature subset selection algorithms for improving patient comfort and reducing energy consumption. Secondly, we introduce an energy model based on a real hardware implementation to determine the change in energy consumption.

O O O O B I I B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O


Semantic and phonetic automatic reconstruction of medical dictations Automatic speech recognition (ASR) has become a valuable tool in large document production environments like medical dictation. While manual post-processing is still needed for correcting speech recognition errors and for creating documents which adhere to various stylistic and formatting conventions, a large part of the document production process is carried out by the ASR system. For improving the quality of the system output, knowledge about the multi-layered relationship between the dictated texts and the final documents is required. Thus, typical speech-recognition errors can be avoided, and proper style and formatting can be anticipated in the ASR part of the document production process. Yet -while vast amounts of recognition results and manually edited final reports are constantly being produced -the error-free literal transcripts of the actually dictated texts are a scarce and costly resource because they have to be created by manually transcribing the audio files.To obtain large corpora of literal transcripts for medical dictation, we propose a method for automatically reconstructing them from draft speech-recognition transcripts plus the corresponding final medical reports. The main innovative aspect of our method is the combination of two independent knowledge sources: phonetic information for the identification of speech-recognition errors and semantic information for detecting post-editing concerning format and style. Speech recognition results and final reports are first aligned, then properly matched based on semantic and phonetic similarity, and finally categorised and selectively combined into a reconstruction hypothesis. This method can be used for various applications in language technology, e.g., adaptation for ASR, document production, or generally for the development of parallel text corpora of non-literal text resources. In an experimental evaluation, which also includes an assessment of the quality of the reconstructed transcripts compared to manual transcriptions, the described method results in a relative word error rate reduction of 7.74% after retraining the standard language model with reconstructed transcripts. After decades of research, speech recognition technology has reached a level where it can be successfully integrated into products for everyday use. In particular, this applies to dictation systems with integrated speech recognition which help reduce the amount of manual transcriptions. In the medical domain, where dictation traditionally plays an important role, speech recognition systems have contributed to a more efficient report creation process since medical transcriptionists no longer have to type whole documents, instead they only do the post-processing to create the final reports. This way, highly skilled medical transcriptionists make better use of their expertise.. In many cases, this post-processing step unfortunately still involves a lot of tedious editing: recognition errors have to be corrected, and the style and formatting of the document have to be adapted to the standards applied to written reports. Particularly for dictations by unexperienced users, post-processing can become time consuming, and thus may lead to many and various deviations between the recognition results and the final reports.. While recognition results and final reports are usually available in abundance, manual transcriptions of the actual spoken words without recognition errors (i.e., assuming perfect recognition) are costly and scarce. For training automatic speech recognition systems, however, literal transcriptions of the actual words are needed.. A standard methodology to overcome the problem of non-literal transcriptions in ASR training is unsupervised or lightly supervised training (Lamel et al., 1998; Kemp and Waibel, 1999) . These approaches allow the generation of statistical models from only small amounts of literal transcriptions together with large amounts of non-literal transcriptions in an iterative fashion. For language model training, methods like linear model interpolation (Chen and Goodman, 1996) or transformation-based learning (Peters and Drexel, 2004) are used to cope with non-literal transcriptions. Although these methodologies lead to reductions in word error rate for as various domains as news broadcasts or transcriptions of class lectures (Hazen, 2006) , they do not give explanations for the mismatches between the non-literal data and the actual wording in the training utterances. Furthermore, problematic segments like disfluencies, hesitations, or speaker corrections cannot be modelled without proper annotation. For these reasons, literal transcriptions are still valuable.. These motivations lead us to the definition of the problem of how a literal transcription can be automatically reconstructed from non-literal transcripts of different information sources. This problem has already been addressed by Pakhomov et al. (2001) for modelling disfluencies and hesitations in medical dictations. However, a comprehensive model for automatic reconstruction needs to go beyond the scope of specific phenomena and provide a generic framework for exploiting the full potential of the analysed documents.. In this article, we propose such a reconstruction framework and describe a system which has been developed for automatically reconstructing the actual spoken words from the recognition result and the final medical reports. These two different input sources are complementary for the task of reconstructing literal transcripts. The resulting reconstructions can be used the same way as manual transcriptions for training speech recognition systems.. The base for reconstruction is an alignment between the written report and the recognition result. The alignment takes into account semantic information (for explaining reformulations) and phonetic information (for explaining recognition errors) as well as syntactic information in terms of document formatting. From the interpretation of the deviations between the written report and the recognition result, the words which are considered to have actually been spoken are reconstructed. According to the proposed methods, we name our approach Semantic and Phonetic Automatic ReConstruction (SPARC). The main innovative aspect of our method is the optimal interplay between two independent knowledge sources, namely semantics and acoustics/phonetics in the categorisation of differences between automatic transcript and final document, as well as in the reconstruction of the original utterance from these two data sources.. Qualitative and quantitative evaluations based on manual transcriptions have shown that, in many cases, the alignment leads to a correct reconstruction. The resulting reconstructed text can serve not only as a base for training and improving the speech recognition system; a deeper understanding of the typical reformulations and reformatting may eventually also support a shift from mere speech recognition to document production in dictation applications.. In the following sections, we will continue with a more detailed account of the SPARC approach in Section 2 and a description of the available text corpora in Section 3. Following this introductory part, in Sections 4, 5, and 6 the three main units of the approach -text alignment, similarity measurement, and text reconstruction -are then elucidated. In Section 7, we report experimental results in terms of the quality of the reconstructed text and an automatic speech recognition experiment with retrained language model. We conclude the paper with a discussion of the results and an outlook for further applications. We have described the SPARC method of semantics and phonetics based similarity measurement for the automatic reconstruction of medical dictations from draft recognised texts and final written reports. The resulting reconstructed texts can be used for various applications in language technology, including but not limited to acoustic and language model adaptation for automatic speech recognition, computer-aided document production in medical transcription, or generally for the development of parallel text corpora of non-literal text resources.. The method is based on an alignment between a draft speech recognition transcript containing errors and a formatted, corrected medical report that may have been paraphrased during the transcription process. The text alignment uses a model of semantic and phonetic similarity to detect corresponding (matching) regions in the texts and to properly align them on multiple levels of segmentation. For this purpose, semantic and phonetic similarity measures were developed for the matching procedure. The resulting alignment is interpreted with a newly developed rule engine which allows precise control over the reconstruction process with context-sensitive reconstruction rules.. The experimental evaluation showed that the text quality improved for the reconstructed text in comparison to both recognised and written text. For recognised texts with a low word error rate, the best reconstruction system improved the F1-score of the best baseline system from 92.9 to 94.1%. In general, phonetics-based rules proved to be more effective than semantic-based rules while semantics-based rules turned out to be more precise. Combining phonetic and semantic knowledge for text reconstruction improved the reconstruction quality. A more detailed analysis revealed that 8% of the resolved mismatches are of pure phonetic nature, only 0.5% of pure semantic nature, and about 17-18% are detectable with both semantic and phonetic measures together. The rule engine for reconstruction proved to have comparable performance to a data-driven classification system for the low word error rate condition, while for medium and high word error rates, the automatic classifiers returned better results.. Concerning the overall benefit for the speech recognition system, an experiment with a retrained language model based on reconstructed texts yielded a word error rate reduction of 7.74% relative in comparison to a standard retraining, and of 4.38% relative for a language model based on randomly reconstructed text. As no specific optimisation has been performed yet, further improvements by parameter tuning are still possible.. In future work, we plan to complement our experiments by evaluating the usage of reconstructed text for retraining of the acoustic models. Furthermore, we want to compare our results to different approaches of including semantic information into the language model, e.g., as classes or embedded grammars.. The focus of the SPARC approach on the assignment of phonetic and semantic similarity between aligned speech recognition results and final reports has turned out to be useful and suitable for the reconstruction of literal transcripts. Three main aspects of the approach have already turned out to be beneficial: (1) reconstructed texts reduce the required amount of manually transcribed texts for training of speech recognition systems. (2) Retraining with reconstructions leads to slightly lower word-error rates in speech recognition. (3) Since the reconstruction and alignment are knowledge based, our methods are already being used as annotation tools for semantic and phonetic information and serve as a starting point for automatic document creation. The results are summarised in Table 8 . Using the reconstructed text language model reduced the overall word error rate from 11.77% to 10.86% which is a relative reduction of 7.74% compared to the baseline classic lan- guage model. The randomly generated as-spoken variants only lead to an overall relative reduction of 4.38%. Table 8 also shows that these improvements are consistent for all speakers. Both, the random as-spoken and reconstructed text language models even outperform the broad language model which was created from substantially more data. Hypothesizing the spoken forms of grammar items is therefore beneficial for the applied 2-stage decoding-rescoring strategy.. Based on the findings from this first experiment, we conducted a second experiment where we gradually increased the corpus size for language model training from 1 million tokens up to the maximum size of about 50 million tokens. Fig. 11 illustrates the evolution of the word-error rate with respect to the size of the language model training corpus. Up to a corpus size of 3 million words, there is not much difference between the models built on randomly generated as-spoken variants and reconstructed text. For a corpus size of 6 million tokens or more, the SPARC method performs consistently better. At 50 million words, the word-error rate begins to go into saturation, so incorporating more data will only have minor effects on the word-error rate.. Apart from the mentioned adjustments, the reconstructed texts were not further optimised or tuned for ASR purposes. Using the SPARC method without any further tuning immediately resulted in the reported improvements. Additional fine-tuning in terms of, e.g., the interpretation of punctuation or the exclusion of leading and trailing irrelevant text blocks in recognised texts may even further improve the performance. The text reconstruction process was evaluated for two different tasks to examine the performance of the SPARC method. First, the quality of the reconstruction was tested. For this test, a literal transcription was reconstructed and compared to a manual reference transcription for a set of medical reports. We define the evaluation as a text retrieval task, because the results reflect how much of the original text can be reconstructed and how much of the reconstructed text is actually part of the original text. This test is a true performance measure of the system, without considering any particular application. The performance of the main components -semantic and phonetic similarity measurement, and text reconstruction -will be evaluated separately in Section 7.1.. Second, the speech recognition performance using reconstructed texts is measured (cf. Section 7.2). In this test, the language model of the speech recogniser producing the recognised texts was re-trained with the reconstructed texts and tested on an independent test set. This test is only an indirect performance measure and is intended to demonstrate the applicability and impact on the speech recognition process. For this reason, we decided to test with a commercially available ASR system instead of an academic one and did not perform specific parameter tuning to keep the results more independent from the actually used ASR system. We chose the domain of Clinical Reports as test case since there was enough data available, both to create reconstructed text (requiring written and recognised text) and to evaluate the performance (requiring reference transcriptions of what has actually been dictated). Our setup reconstructed 101,607 reports from 504 authors (between 45 and 507 reports per author) running in parallel on four standard personal computers for three weeks and produced a text corpus of 52 million words (cf. Table 7), the equivalent of about 9,000 hours of sound.. The SPARC reconstruction approach was selected as follows: The baseline word error rate of the chosen corpus indicated a low-to-medium WER condition. According to Table 6 either the rule-based system with I+P+S rules or one of the data-driven reconstruction approaches appeared to be promising for text reconstruction. Previous language model adaptation experiments conducted on less data showed that a certain fragmentation is introduced in the reconstructed text by the data-driven reconstruction approaches resulting in higher language model perplexities (Petrik and Pernkopf, 2008b) . Therefore, we selected the rule-based I+P+S system for text reconstruction in the experiments.. The speaker-independent acoustic models of the ASR system have been trained on 200 h (female speakers) and 300 h (male speakers) of acoustic material recorded on a telephone channel with 4 kHz bandwidth, Acoustic speaker-adaptation was performed using Maximum-Likelihood Linear Regression (MLLR) for the first 15 min of sound, and Maximum-A-Posteriori (MAP) adaptation for the rest of the available data, which was 10 h of sound for each speaker in the test set.. For this domain, a large medical lexicon of 58,103 words was used, giving a high coverage on both the test set and the training corpus (OOV rates <1.5%).. Recognition tests were performed on a set of 239 reports from 2 female and 3 male authors (3 h of sound per user), all in the domain of Clinical Reports, 4 and all recorded through a telephone channel with 4 kHz bandwidth. The best available acoustic references for these speakers were used. The baseline word error rate on our test set was 11.77% (cf.

O O O O O O O O B I I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


The Architecture and Performance of a Stochastic Competitive Evolutionary Neural Tree Network A new dynamic tree structured network-the Stochastic Competitive Evolutionary Neural Tree (SCENT) is introduced. The network is able to provide a hierarchical classification of unlabelled data sets. The main advantage that SCENT offers over other hierarchical competitive networks is its ability to self-determine the number and structure of the competitive nodes in the network without the need for externally set parameters. The network produces stable classificatory structures by halting its growth using locally calculated, stochastically controlled, heuristics. The performance of the network is analysed by comparing its results with that of a good non-hierarchical clusterer, and with three other hierarchical clusterers and its non stochastic predecessor. SCENT's classificatory capabilities are demonstrated by its ability to produce a representative hierarchical structure to classify a broad range of data sets. In this paper we discuss a novel neural network model that is capable of discovering hierarchical structure and categories from unlabelled data.. Simple competitive neural networks are straightforward methods for finding structured knowledge, in that any inherent clustering in a set of data is discovered. The process of discovery is, however, limited to the discovery of a predefined number of clusters. A dynamic network is set a significantly more difficult problem: to find not only the clusters but also their number, by dynamically creating classifying neurons in response to apparent structure in the data. A still more difficult task in the discovery of structured knowledge is the problem of finding the hierarchical clustering structure in a data set. Here, natural clusters, their number, and their hierarchical relationship are sought. Moreover if such a hierarchy can be found then the codevectors of the network can be considered as prototypes for the categories they classify, see the discussion in Section 4.2.. Competitive Learning Neural Networks (CLNNs) [1] classify unlabelled data sets, providing single layer cluster analysis. The nature of the classification found by a CLNN is strongly influenced by the number of competitive nodes used, which must be decided upon before starting. As CLNNs produce single layer classifications they are not suitable for use in application areas such as biological taxonomy [2] that require a hierarchical analysis of the data.. Recent research [3] [4] [5] [6] [7] has investigated the possibility that the nodes could be arranged in a tree structure; such an arrangement has two potential benefits: searching a tree is fast and any hierarchical information in the data may be explicitly represented in the tree. Most clustering algorithms, neural net or otherwise, expect the number of classes used to be predefined; this is an obvious problem if no a-priori knowledge about the data is available. Consequently further research has been directed towards exploring the possibility that a network can dynamically evolve the appropriate structure in response to the data. Such an approach naturally leads to tree structures-a unit may produce child nodes if it feels it is not classifying its local data with sufficient granularity.. Dynamic Neural Tree Networks (DNTNs) [4, 8] attempt to combine the advantages of using a tree structure and of growing the network, these networks are reviewed in Section 2. DNTNs show promise in their ability to produce stable yet flexible hierarchical structures. The nature of the hierarchical structure and the quality of the final classification produced by the networks is, however, very strongly influenced by the values given to externally set parameters. A new network, the Competitive Evolutionary Neural Tree (CENT) [9] , was developed to extend the ideas of DNTNs.. Most unsupervised neural networks perform some form of gradient descent on an error function, normally the sum squared error, and as such are likely to fall into a local minimum. Attempts to overcome this problem have concentrated on two approaches: modifying the objective function and thereby the update rule, and secondly using some form of simulated annealing [10] . In practice, in both approaches, all the nodes in the network are updated on each vector presentation, rather than just the winner. The term soft competition is used to embrace both methods.. One of the most effective models that uses soft competition in a non-hierarchical clusterer is the Neural Gas model [11] . Here all nodes are updated at each iteration with the competition becoming progressively less soft over time, with the network eventually approaching the standard single winner CLNN paradigm. The success of the simulated annealing approach, as exemplified in Neural Gas, encouraged the exploration of mediated stochasticity within the context of CENT, which produced a more robust model Stochastic CENT, SCENT, introduced in Section 3.. For the purposes of the results reported here we have compared our model with the two other dynamic neural trees, the dynamic competitive learning technique of Racz and Klotz [4] , and the Neural Tree of Li [8] and with the best of the non-hierarchical clusterersthe Neural Gas model. We have also compared the representational ability of our SCENT network with another hierarchical neural clusterer-the HART network of Bartfai [12] . Results of these comparisons can be found in Section 4. Using neural nets to perform data exploration, and thereby discover any structural knowledge implicit within the data is difficult; most models require the pre-imposition of a maximum number of clusters, and will normally classify the data to utilise all classificatory units. Some recent architectures have attempted to dynamically create tree structures to overcome the need for prescribing cluster number and to give a hierarchical view of the data. Their use though has been problematic, with a tendency to great sensitivity to multiple initial parameterisation. SCENT attempts to overcome this problem by being completely autonomous. Both CENT and SCENT produce consistent tree structures across a range of data sets. SCENT exhibits more stability then CENT with hierarchical data due to low variance.. The SCENT model has produced a consistently good performance over all of the data sets presented so far. The SCENT performance has been better than that of the two original DNTN networks over almost all the tests. In test 3 the Li performance was excellent and outperformed the SCENT, but the Racz and Klotz performance was significantly worse than the SCENT model. The SCENT performance has been good over both uniform data sets, where the Racz and Klotz network performs poorly, and data sets with ranges of cluster densities and sizes, that cause the Li network to perform poorly. The consistency of the SCENT performance over all types of data is notable and shows that the methods used within the model to assess each data set are both thorough and flexible.. The fact that the SCENT network has been able to produce performances that are close in terms of SSE to the Neural Gas network is very encouraging. It is harder for a dynamic network, particularly a tree structured network, to produce a good SSE result than it is for a flat static network such as the Neural Gas. The ability of the SCENT to provide the advantages given by the tree structure whilst maintaining good SSE results is a positive result.. From the results and analysis presented in Section 4.2 it can be seen that the SCENT model is capable of discovering interesting hierarchical structure in unlabelled data, as well as providing a straightforward codevector reduction. The SCENT model has succeeded in producing a hierarchical tree structure reflecting the clear visual differences between the images and is capable of learning stable hierarchical clusters that include both super and sub categories from semantic data.. The development of classification hierarchy models such as HART and SCENT allows for rapid high level analysis of data sets. Critical features of the data are easily identified, as are any features of little statistical significance, at a number of levels within any natural hierarchy of the data set. The top-down approach of SCENT allows for the development of flexible prototype clusters based on hierarchical code vectors of an unlabelled data set.. The full theoretical relationship between training data, in general, and the resulting structures produced by SCENT is an issue currently being investigated. The performance of CENT and SCENT as hierarchical clusterers, particularly their ability to discover hierarchical structure, has been evaluated. Both models were evaluated comparatively (Section 4.1) allowing an empirical judgment of the ability of the two networks to be made over a range of data sets. Results of CENT and SCENT were compared with each other and with the three other neural models described in this paper: Neural Gas [11], Li's Neural Tree [20] and Racz and Klotz's dynamic learning model [4] . SCENT was also evaluated structurally (Section 4.2) using two further data sets to determine the representational aspects of SCENT's clustering capabilities. A brief comparison of SCENT with HART [12] was also made for the second of these data sets. Cluster analysis is the production of a classification scheme for initially unclassified and unlabelled data; the aim being that the classification produced should represent the natural groups/clusters that may be present in the data.. Unsupervised clustering using neural networks is a well established technique, with the simple competitive learning neural network (CLNN) being the elementary model and Kohonen's self organising maps (SOMs) the most popular manifestation [1] .. In a simple competitive network the weight vectors of the nodes in the output layer of the network after training should ideally represent the prototypes of the clusters found in the data set. The initial values for these prototypes-the weight vectors-are normally set to random values in an attempt to avoid bias. Each input from the data set is then presented to the network in turn and output layer nodes compete for the right to classify the input and update their weight vectors. The winner of the competition is the node whose weight vector is most similar to the input vector presented. If the winning node's weight vector is w i * then:. where x is the input vector and w i are the prototype weight vectors. The winning node has its weight vector updated to make it more similar to the input that caused it to win according to:. where α is the learning rate of network, normally in the range 0.01 to 0.15, and n is the time step.. The output nodes gradually move towards the centre of the group of inputs which they classify, the node's weight vector becoming the prototype for the cluster of inputs it classifies. In this way the clusters that exist in a data set can be discovered. The network is iteratively reducing the Sum of Squared Error (SSE) by applying (2) to the winning node, where SSE is defined by:. where c is the number of clusters, x are the input vectors, X i is the ith cluster and m i is the mean vector in X i .. The behaviour of the network is very similar to that of the statistical technique of k-means clustering. If the output nodes are updated in batch mode, that is updated once per presentation of the whole data set, and dead units (units that are initialised too far from the data and never win the competition) are prevented then the CLNN has been shown to be functionally equivalent to k-means clustering [13] .. In a SOM the classifying units are arranged with some spatial topology, usually a grid. The most common form of ordering for a SOM is a two dimensional lattice providing a two dimensional output space. The winning node for each input is found and updated in the same way as for a CLNN. However, in a SOM the neighbours of the winning node are also updated, where the neighbours of a node are defined by the ordering imposed on the output nodes. Typical neighbours for a two dimensional lattice are contained within a square grid centred on the winning unit. This neighborhood updating means that neighboring nodes in the output space are pulled into the same input space area as the winning node, thus producing an ordered mapping from the input space to the output nodes. The result is that inputs from similar parts of the input space should be classified by nodes from similar parts of the output lattice.. The SOM provides a classification that, because of the structure imposed on the output nodes, is easier to interpret meaningfully than the CLNN. The movement of the neighborhood nodes lessens the effect on the SOM of some of the problems met by the CLNNs, specifically dead units.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O