Failure recovery for structured p2p networks: Protocol design and performance under churn Measurement studies indicate a high rate of node dynamics in p2p systems. In this paper, we address the question of how high a rate of node dynamics can be supported by structured p2p networks. We confine our study to the hypercube routing scheme used by several structured p2p systems. To improve system robustness and facilitate failure recovery, we introduce the property of K-consistency, K P 1, which generalizes consistency defined previously. (Consistency guarantees connectivity from any node to any other node.) We design and evaluate a failure recovery protocol based upon local information for K-consistent networks. The failure recovery protocol is then integrated with a join protocol that has been proved to construct K-consistent neighbor tables for concurrent joins. The integrated protocols were evaluated by a set of simulation experiments in which nodes joined a 2000-node network and nodes (both old and new) were randomly selected to fail concurrently over 10,000 s of simulated time. In each such ''churn'' experiment, we took a ''snapshot'' of neighbor tables in the network once every 50 s and evaluated connectivity and consistency measures over time as a function of the churn rate, timeout value in failure recovery, and K. We found our protocols to be effective, efficient, and stable for an average node lifetime as low as 8.3 min. Experiment results also show that the average routing delay of our protocols increases only slightly even when the churn rate is greatly increased. Structured peer-to-peer networks are being investigated as a platform for building large-scale distributed systems [8, 10, 12, 14] . The primary function of these networks is object location, that is, mapping an object ID to a node in the network. For efficient routing, each node maintains O(log n) pointers to other nodes, to be called neighbor pointers, where n is the number of network nodes. To locate an object, the average number of application-level hops required is O(log n). Each node stores neighbor pointers in a table, called its neighbor table. The design of protocols to construct and maintain ''consistent'' neighbor tables for network nodes that may join, leave, and fail concurrently and frequently is an important foundational issue.. Of interest in this paper is the hypercube routing scheme used to achieve scalable routing in several proposed systems [8, 10, 14] . Our first objective is the design of a failure recovery protocol for nodes to re-establish consistency of their neighbor tables after other nodes have failed. 2 Neighbor table consistency guarantees the existence of at least one path from any source node to any destination node in the network [7] . Such consistency however may be broken by the failure of a single node. To increase robustness and facilitate the design of failure recovery, we introduce K-consistency, K P 1, which generalizes consistency previously defined [7] . We design and evaluate a failure recovery protocol, which includes recovery from voluntary leave as a special case, for K-consistent networks. The protocol was found to be highly effective for K P 2. From 2080 simulation experiments in which up to 50% of network nodes failed at the same time, we found that all ''recoverable holes'' in neighbor tables due to failed nodes were repaired by our protocol for K P 2, that is, the neighbor tables recovered K-consistency after the failures in every experiment for K P 2. Furthermore, the vast majority of the holes in neighbor tables were repaired with no communication cost. The protocol uses only local information at each node and is thus scalable to a large n.. Our second objective is integration of the failure recovery protocol with a join protocol that has been proved to construct K-consistent neighbor tables for an arbitrary number of concurrent joins in the absence of failures and also shown to be scalable to a large n [7, 3] . Such integration requires extensions to both the failure recovery and join protocols. For a network with concurrent joins and failures, the failure recovery protocol needs to distinguish between nodes that are still in the process of joining, called T-nodes, and nodes that have joined successfully, called S-nodes. The join protocol, on the other hand, needs to be extended with the ability to invoke failure recovery and to backtrack. Furthermore, when a node is performing failure recovery, its replies to some join protocol messages must be delayed. We ran 980 simulation experiments in which the number of concurrent joins and failures was up to 50% of the initial network size. We found that, for K P 2, our protocols constructed and maintained K-consistent neighbor tables after the concurrent joins and failures in every experiment.. Our third objective is to explore how high a rate of node dynamics can be sustained by the integrated protocols for hypercube routing. We performed a number of (relatively) long duration experiments, in which nodes joined a 2000-node network at a given rate, and nodes (both existing and joining nodes) were randomly selected to fail concurrently at the same rate. In each such churn experiment, we took a snapshot of neighbor tables in the network once every 50 simulation seconds and evaluated network connectivity and consistency measures over time as a function of the churn rate, timeout value in failure recovery, and K. Our protocols were found to be effective, efficient, and stable for a churn rate up to four joins and four failures per second. By Little's Law, the average lifetime of a node was 8.3 min at this rate. For comparison, the median lifetime measured for Gnutella and Napster was 60 min [11] .. We also found that, for a given network, its sustainable churn rate is upper bounded by the rate at which new nodes can join the network successfully (become S-nodes). We refer to this upper bound as the network's join capacity. We found that a network's join capacity decreases as the network's failure rate increases. For a given failure rate, we found two ways to improve a network's join capacity: (i) use the smallest possible timeout value in failure recovery, and (ii) choose a smaller K value. Since improving a network's join capacity improves its sustainable churn rate, our observation that a smaller K (less redundancy) leads to a higher join capac-ity is consistent with the conclusion in [1] . Furthermore, we found that a network's maximum sustainable churn rate increases at least linearly with n (the number of network nodes) for n from 500 to 2000. This validates a conjecture that our protocols' stability improves as the number of S-nodes in the network increases. Experiment results also show that our protocols, by striving to maintain K-consistency, were able to provide pairwise connectivity higher than 99.9995% (between S-nodes) at a churn rate of two joins and two failures per second for n = 2000 and K = 3. Furthermore, the average routing delay increased only slightly even when the churn rate was greatly increased.. The balance of this paper is organized as follows. In Section 2, we present an overview of the hypercube routing scheme and define K-consistency. In Section 3, we describe our failure recovery protocol and present results from 2080 simulation experiments. In Section 4, we present our join protocol that has been proved to construct and maintain Kconsistent networks for concurrent joins. In Section 5, we describe how to extend the join and failure recovery protocols to handle concurrent joins and failures and present results from 980 simulation experiments. In Section 6, we present results from long-duration churn experiments in which nodes join and fail continuously. In Section 7, we investigate the routing performance of our protocols under different churn rates. We discuss related work in Section 8 and conclude in Section 9. For structured p2p networks that use hypercube routing, we introduced the property of K-consistency and designed a failure recovery protocol for K-consistent networks. The protocol was evaluated with extensive simulations and found to be efficient and effective for networks of up to 8000 nodes in size. Since our protocol uses local information, we believe that it is scalable to networks larger than 8000 nodes.. The failure recovery protocol was integrated with a join protocol that has been proved to construct Kconsistent networks for concurrent joins and shown analytically to be scalable to a large n [3] . From extensive simulations, in which massive joins and failures occurred at the same time, the integrated protocols maintained K-consistent neighbor tables after the joins and failures in every experiment for K P 2.. From a set of long-duration churn experiments, our protocols were found to be effective, efficient, and stable up to a churn rate of four joins and four failures per second for 2000-node networks (with K = 2 and 5-s timeout). By Little's Law, the average node lifetime was 8.3 min. We discovered that each network has a join capacity that upper bounds its join rate. The join capacity decreases as the failure rate increases. For a given failure rate, the join capacity can be increased by using the smallest timeout value possible in failure recovery or by choosing a smaller K value.. We also observed from simulations that our protocols' stability improves as the number of S-nodes increases. More specifically, for 500 6 n 6 2000, we found that a network's maximum sustainable churn rate increases at least linearly with network size n. The trend in our simulation results suggests that as network size increases beyond 2000 nodes, the average node lifetime can be less than 12.1 min for K = 3 and 8.3 min for K = 2. Furthermore, the average number of routing hops remains at log b n for networks under heavy churn.. The storage and communication costs of our protocols were found to increase approximately linearly with K (see Section 7 in [4] ). The results in this paper show that the network robustness improvement is dramatic when K is increased from 1 to 2. We believe that p2p networks using hypercube routing should be designed with K P 2. However, a bigger K value results in higher storage and communication overhead; and as shown in the churn experiments, a large K also reduces the join capacity of a network. Thus, for p2p networks with a high churn rate, we recommend a K value of 2 or at most 3. For p2p networks with a low churn rate, K may be 3 or higher (say 4 or 5) if additional route redundancy is desired.. Our integrated protocols for join and failure recovery in this paper have been implemented in a prototype system named Silk [3] . Aside from being extremely effective, our failure recovery protocol is also very efficient because recoverable holes repaired in step (a) incur no communication cost, while each hole repaired in step (b) incurs a communication cost of at most 2(K À 1) messages. Table 3 shows that, for K P 2, the majority of recoverable holes were repaired in step (a) and almost all of them were repaired by the end of step (b). Note that if a recoverable hole is repaired in step (a), its recovery time is (almost) zero. The time required for each subsequent step ((b)-(d)) is at most the step's timeout value. For the timeout value of 20 s per step, the average time to repair a recoverable hole was less than 5.88 s for b = 16, d = 40, and K = 3 in Table 3 . For a timeout value of 5 s per step, the average time to repair a recoverable hole was found to be less than 1.45 s for b = 16, d = 40, and K = 3 from a different set of experiments. Table 4 shows the total number of holes, the number of irrecoverable holes, as well as the number of recoverable holes repaired at each step for the same simulation experiments shown in Table 3 . Observe from Table 4 that when K was increased, Table 3 Cumulative fraction of recoverable holes repaired by the end of each step, n = 4000, f = 800. Step (a). Step (b). Step (c). Step ( even though the total number of holes increased, the number of recoverable holes repaired in step (b) did not increase much with K; the number of holes repaired actually declined in steps (c) and (d). Thus while increasing K causes the number of recoverable holes repaired in step (a) to increase, these repairs are performed with zero communication cost. Nevertheless, the communication cost of failure recovery increases with K because the number of irrecoverable holes increases with K. Note that for each irrecoverable hole, all four steps of failure recovery are executed.. . Fig. 10 summarizes our results, which are plotted versus median node lifetime along the horizontal axis. A smaller median node lifetime corresponds to a higher churn rate. Hence, in each figure, churn rate increases from right to left. Fig. 10(a) shows the percentage of successful routing tests. Fig. 10(b) shows the average number of hops from source to destination over successful routing tests. In the source-duplication and backtracking approach, for each routing test, we used the number of hops traveled by the message that arrived at the destination first. Fig. 10(c) shows the average delay over successful routing tests.. Observe from Fig. 10 (a) that with backtracking only, the percentage of successful routing is already very close to 100%. With the addition of sourceduplication, the success percentage becomes even closer to 100% (the percentage was in fact 100% for all median lifetimes greater than or equal to 46.2 min).. Also observe from Figs 10(b) and (c), when the median node lifetime decreases (from right to left), the average number of hops and average routing delay increase very slightly. Each such increase is due to a small increase in backtracking occurrences when node failures become more frequent. In particular, the average number of hops for all lifetimes of both curves in Fig. 10(b) is within the range of 2.275 to 2.496, and actually less than log 16 (2000), which is 2.74. This confirms our conjecture that by striving to maintain K-consistency in neighbor tables, our protocols preserve scalable routing in the hypercube routing scheme even in the presence of heavy churn.. Lastly, from Figs. 10(b) and (c), observe that the addition of source-duplication to backtracking provides only a small improvement in the average number of hops and routing delay. Among related work, both Pastry [10] and Tapestry [14] make use of hypercube routing. Pastry's approach for failure recovery is very different from the one in this paper. In addition to a neighbor table for hypercube routing, each Pastry node maintains a leaf set of 32 nearest nodes on the ID ring to improve resilience. Leaf set membership is actively maintained. Pointers for hypercube routing, on the other hand, are used as shortcuts and repaired lazily. Tapestry's basic approach for failure recovery is similar to ours in that it also stores multiple nodes in a neighbor table entry. However, the property of K-consistency is not defined and thus not enforced in Tapestry. Furthermore, Tapestry's join and failure recovery protocols are based upon use of a lower-layer Acknowledged Multicast protocol supported by all nodes [2] . Our protocols do not require such reliable multicast support and are very different from the Tapestry protocols. Recently, two other papers also addressing the problem of churn in structured p2p networks were published. Li et al. [6] used a single workload to compare the performance of four routing algorithms under churn. In their experiments, the churn rate was fixed with the corresponding average node lifetime equal to 60 min. Their goal was to study the impact of algorithm parameter values on system performance, more specifically, the tradeoff between routing latency and bandwidth overhead.. Rhea et al. [9] identified and evaluated three factors affecting DHT performance under churn, namely: reactive versus periodic failure recovery, algorithm for calculating timeout values, and proximity neighbor selection. They have also investigated the impact of a wide range of churn rates on average routing delay (called lookup latency in their paper) as the performance measure for several DHTs.. We have a different set of objectives in this paper. Our first objective was the design of a failure recovery protocol based upon local information for hypercube routing and its integration with a join protocol to maintain K-consistency of neighbor tables. We use a stronger definition of consistency (for neighbor tables) than the consistency definition (for lookups) used in [9] . In addition to the impact of churn rate on average routing delay, we also evaluated the impact of churn rate on neighbor table consistency and pairwise node connectivity provided by the neighbor tables. Furthermore, we explored the notion of a sustainable churn rate and found that it is upper bounded by the rate at which new nodes can join the network successfully. We refer to this upper bound as the join capacity of a network. We found two ways to improve a network's join capacity, namely, by using the smallest possible timeout value and choosing a smaller K value. Fig. 10 (c) in this paper for 3-consistent hypercube routing can be compared to Figs. 7 and 9 in [9] for Bamboo and Chord. In each figure, average routing delay is plotted versus median node lifetime (same as median session time in [9] ). Consider and compare the shapes of the average routing delay graphs (ignore the absolute delay values since different topologies and link delays were used in different experiments). Observe that when the median node lifetime decreases, the average routing delay increases much more significantly for Chord and also Bamboo than for 3-consistent hypercube routing. Such performance degradation is due to the different failure recovery strategies used in Bamboo and Chord. In Bamboo, which follows Pastry, neighbors in a node's leaf set are actively maintained while neighbors in the node's hypercube routing table are repaired lazily. As stated in [9] , ''the leaf set allows forward progress (in exchange for potentially longer paths) in the case that the routing table is incomplete.'' Thus, when failures happen more and more frequently during periods of high churn, the average routing delay of Bamboo increases much more than in a hypercube routing scheme that strives to maintain K-consistency of its routing tables. Fig. 10(b) shows that in our experiments the average number of hops remained at approximately log b n for the entire range of churn rates (node lifetimes). To evaluate the performance of our failure recovery protocol, 2080 simulation experiments were conducted on our own discrete-event packet-level simulator. We used the GT_ITM package [13] to generate network topologies. For a generated topology with a set of routers, n nodes (end hosts) were attached randomly to the routers. For the simulations reported in Table 2 , three topologies were used. The 1000-node and 2000-node simulations used a topology with 1056 routers. The 4000-node simulations used a topology with 2112 routers. The 8000-node simulations used a topology with 8320 routers. We simulated the sending of a message and the reception of a message as events, but abstracted away queueing delays. The end-to-end delay of a message from its source to destination was modeled as a random variable with mean value proportional to the shortest path length in the underlying network. 5 In each simulation, a network of n nodes with Kconsistent neighbor tables was first constructed. Then a number, f, of randomly chosen nodes failed. For 1000-node and 8000-node simulations, the f nodes failed at the same time. For 2000-node simulations and each specific K value, the f nodes failed at the same time for 84 out of the 180 experiments; a Poisson process was used to generate failures in the balance of the experiments, with half of the experiments at the rate of 1 failure per second and the other half at the rate of 1 failure every 10 s. For comparison, the timeout value used to determine whether a neighbor has failed was 5 s, and the timeout value used in each of the protocol steps (b)-(d) was 20 s. Therefore, most failure recovery processes ran concurrently even when the Poisson rate was slowed to one failure every 10 s. For 4000-node experiments and each specific K value, the f nodes failed at the same time in 104 out of the 116 experiments, with a Poisson process at the rate of 1 failure per second used in the balance of the experiments.. We conducted simulations for different combinations of b, d, K, n and f values. For each network of n nodes, n 2 {1000, 2000, 4000, 8000}, four pairs of (b, d) were used, namely: (4, 16), (4, 64) , (16, 8) , and (16, 40). 6 Then, for each (b, d) pair, K was varied from 1 to 5. For each (n, b, d, K) combination, f was varied from 0.05n to 0.1n, 0.15n, 0.2n, 0.3n, 0.4n, and 0.5n (1540 experiments were run for Table 2 Results from 2080 simulation experiments (f was 0.05n, 0.1n, 0.15n, 0.2n, 0.3n, 0.4n or 0.5n). Number of simulations 5 The maximum end-to-end delay in 8000-node simulations was 969 ms. 6 In Tapestry To construct the initial K-consistent networks for simulations, we experimented with four approaches to choose neighbors for each entry: (i) choose K neighbors randomly from qualified nodes, (ii) choose K closest neighbors from qualified nodes, (iii) choose K neighbors randomly from qualified nodes that are within a multiple of the closest neighbor's distance, (iv) use our join protocol in Section 4 to construct a K-consistent network. We conjecture that a K-consistent network constructed by approach (iii) would be closest to a real network whose neighbor tables have been optimized by some heuristics. As shown below, we found that for K P 2, our failure recovery protocol was very effective irrespective of the approach used for initial network construction. (All four approaches were used for different experiments in the set of 2080 experiments.) Table 2 shows a summary of results from the 2080 simulation experiments. In a simulation, if all recoverable holes are repaired (thus K-consistency recovered) at the end of the simulation, it is recorded as a perfect recovery in Table 2 . In the 2080 simulation experiments, every simulation for K P 2 finished as a perfect recovery, i.e., every recoverable hole was repaired with a qualified substitute. Thus in K-consistent networks, for K P 2, our failure recovery protocol is extremely effective. Table 3 presents results from ten simulations for a network with 4000 nodes and 800 failures, where the initial neighbor tables were constructed using approach (iii), described above. The results show the cumulative fraction of recoverable holes that were repaired by the end of each step in the recovery protocol. For instance, for the simulation with parameters b = 4, d = 64 and K = 2, more than 66.8% of recoverable holes were repaired by the end of step (a), 93.8% were repaired by the end of step (b), 99.8% were repaired by the end of step (c), and all were repaired by the end of step (d). From Table 3 , observe that step (d) in our recovery protocol was rarely used. There was a dramatic improvement in the recovery protocol's performance when K was increased from 1 to 2. Also observe that the fraction of recoverable holes that were repaired after each step increases with K.

B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Secure, dependable and publicly verifiable distributed data storage in unattended wireless sensor networks Abstract In unattended wireless sensor networks (UWSNs), sensed data are stored locally or at designated nodes and further accessed by authorized collectors on demand. This paradigm is motivated by certain scenarios where historical or digest data (e.g., average temperature in a day), instead of real-time data, are of interest. The data are not instantly forwarded to a central sink upon sensing, thereby saving communication energy for transmission. Such a paradigm can also improve data survivability by making use of distributed data storage in cheap flash memory on nodes. However, the security and dependability of such data storage are critical for the future data accessibility in UWSNs. To address this issue, we propose a secure and dependable distributed storage scheme for UWSNs. Our scheme takes advantages of both secret sharing and Reed-Solomon code, which can achieve computational security and maintain low communication overhead in terms of shortened data dispersing size. We also propose a general coding method to publicly verify data integrity in a distributed manner, with low communication and storage overhead, and without the need of holding original data. The extensive analysis justifies that our scheme can provide secure, dependable and publicly verifiable distributed data storage in UWSNs even in the presence of node compromise and Byzantine failure. Wireless sensor networks (WSNs) consist of a large number of sensor nodes that can be easily deployed to various environments. The sensor nodes usually generate a large amount of data over their lifetime to report or record the environmental context. Data storage in WSNs mainly falls into two categories, namely centralized data storage and distributed data storage [1] . In the former case, data are sensed, processed, aggregated and managed at a central location, usually a sink. In the latter case, after a sensor node has generated some data, the node stores the data locally or at some designated nodes within the network, instead of immediately forwarding the data to a centralized location out of the network [2] [3] [4] . Such a paradigm is called "Unattended WSNs" (UWSNs) [2] . UWSNs paradigm is motivated by scenarios whereby not real-time information, but digest information is of interest. The digest information includes historical summarization or locally extracted results on data processing, such as the average temperature during last three months; the highest and lowest humidity degree during last 24 h; or more specifically, the average concentration of a chemical element in soil during last half year [5] . The digest information is provided on demand upon user's retrieval, avoiding frequent data transmission to sinks. It thus remarkably saves the communication energy. Moreover, there exist some scenarios that may not be convenient for deploying robust real-time sinks, e.g. in the sea, volcano, or places far away from collectors. The data have to be stored locally or in designated nodes in the network and wait for further collection.. Security and dependability of the stored data in UWSNs are of paramount importance for their functionality. The reasons are listed as follows: 1) As sensor nodes are exposed to unattended environment, data may be lost due to failure of sensor nodes, such as power depletion, melting, corroding or getting smashed; 2) data may be stolen or modified by mobile adversaries [6] that can compromise a selective subset of sensor nodes. Hence, a fault-tolerant and compromise-resilient scheme for distributed data storage has to be in position to ensure data availability, integrity and confidentiality.. The straightforward data redundancy or fault-tolerant methods for data availability cannot be applied in UWSNs domain that has resource constraints. Besides, a naive way to check the data integrity of distributed data by hash value is also not appropriate. More precisely, the hash values of the distributed data are stored at an integrity checker or verifier, and the checker sends requests to data holders to launch a check. If the computed hash values of returned data do not match stored hash values, the data are modified, thus obviously incurring high communication overhead due to the overwhelming data communications. It also induces remarkable storage overhead as the checker has to store a large volume of hash values. Especially, such a checking method is not reliable as the checker is a single point of failure. Hence, a lightweight method for data integrity check with distributed public verification is mandatory.. Currently, UWSNs paradigm, especially its security issues attract more and more attentions in research communities [2-5, 7, 8] . Nevertheless, security, dependability, and integrity verifiability of distributed data storage still require thorough exploration and tailored design.. Objective of the paper: The goal of this paper is to investigate techniques for achieving the secure and dependable distributed data storage as well as lightweight and public verifiability of integrity for UWSNs.. Contributions: 1) In this paper, we propose a secure and dependable data distributed storage scheme by making use of secret sharing and Reed-Solomon code. Such a scheme is resilient for random fault and node compromise, with communication and storage efficiency maintained. 2) We present a generic technique called two granularity linear code (TGLC), which can verify distributed data's integrity publicly and efficiently, i.e., in a distributed manner and with low communication and storage overhead without original data.. The remainder of the paper is organized as follows: The network assumption, adversary model and design goals are presented in section 2. Section 3 presents some preliminaries on secret sharing and error correcting code. Section 4 proposes the data storage scheme. The extensive performance analyzes and security analysis are given in section 5. Section 6 proposes a lightweight coding method for public verifiability of data integrity. Finally, section 7 concludes the paper. In this paper, we have proposed a secure and dependable data distributed storage scheme for in-networking stored sensing data in UWSNs. To achieve the maximum of accessible data upon retrieval, we take advantages of secret sharing and Reed-Solomon code, without loss of computational security and with low communication and storage overhead. The extensive analysis on performance, security and dependability verified that our scheme performs well even in the presence of node compromise and Byzantine failure.. Besides it remains efficient in terms of storage, communication and computation owing to shortened distributed data size.. Furthermore, we presented a generic coding method called Two Granularity Linear Code to efficiently check data integrity. It consists of Intra-code and Inter-code that can be computed commutatively. Such a property provides inner linkage of distributed data via digests and parities with small size. It can be applied to verify data integrity publicly in a distributed manner without original data. By making use of TGLC, we have thus proposed a lightweight scheme for verifying distributed data integrity. The extensive analysis has verified that the false positive of checking result is zero and the false negative vanishes exponentially with appropriate coding parameters and checking rounds.

O O O O O O O O O O B I I O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O


Site-Based Partitioning and Repartitioning Techniques for Parallel PageRank Computation Abstract-The PageRank algorithm is an important component in effective web search. At the core of this algorithm are repeated sparse matrix-vector multiplications where the involved web matrices grow in parallel with the growth of the web and are stored in a distributed manner due to space limitations. Hence, the PageRank computation, which is frequently repeated, must be performed in parallel with high-efficiency and low-preprocessing overhead while considering the initial distributed nature of the web matrices. Our contributions in this work are twofold. We first investigate the application of state-of-the-art sparse matrix partitioning models in order to attain high efficiency in parallel PageRank computations with a particular focus on reducing the preprocessing overhead they introduce. For this purpose, we evaluate two different compression schemes on the web matrix using the site information inherently available in links. Second, we consider the more realistic scenario of starting with an initially distributed data and extend our algorithms to cover the repartitioning of such data for efficient PageRank computation. We report performance results using our parallelization of a state-of-the-art PageRank algorithm on two different PC clusters with 40 and 64 processors. Experiments show that the proposed techniques achieve considerably high speedups while incurring a preprocessing overhead of several iterations (for some instances even less than a single iteration) of the underlying sequential PageRank algorithm. Sparse matrix partitioning models are shown to be quite successful in load balancing and minimizing the total volume of communication during the repeated parallel SpMxV in PageRank computations. However, the vast sizes of the web matrices and the high-preprocessing overhead incurred by these models render this solution infeasible in practice. As a remedy, this paper investigated several sitebased compression schemes and sparse matrix partitioning algorithms, targeting to obtain acceptable preprocessing overheads compared to the previously used page-based schemes. The results indicate that with similar load balance values and total communication volumes achieved, preprocessing overheads can be drastically reduced in sitebased schemes. Among the several schemes tested, columnwise partitioning together with page-to-site compression is found to achieve much better performance. This can be attributed to the higher similarity among outlink patterns of pages than the inlink patterns of pages. We should note here that this finding slightly contradicts with the findings in a recent work [10] , and this discrepancy requires further study. Furthermore, the proposed matrix repartitioning algorithms, which handle the data migration costs arising due to the already distributed nature of web matrices are also tested on very large scale real-world data and found to yield good speedups with scalable overheads.. As future work, we are planning to investigate the following issues: As the LAW codes minimize storage requirement through compression, the application and adaptation of the parallelization models and methods in this paper can be applied for efficient parallelization of LAW codes for the solution of large problems on parallel systems with relatively small memory. As the recursive lumping-based reordered PageRank algorithm of [45] reduces the size of the iteration matrix significantly, it should be investigated for efficient parallelization as well. The models and methods proposed in this work apply to the efficient parallelization of the first subsystem of [45] . However, the parallelization of the numerous SpMxV operations to be performed for finding the remaining subvectors needs research effort because of the highly sequential nature of the forward substitution scheme. Cevdet Aykanat received the BS and MS degrees from the Middle East Technical University, Ankara, Turkey, both in electrical engineering, and the PhD degree in electrical and computer engineering from Ohio State University, Columbus. He was a Fulbright scholar during the PhD studies. He worked at the Intel Supercomputer Systems Division, Beaverton, Oregon, as a research associate. Since 1989, he has been affiliated with the Department of Computer Engineering, Bilkent University, Ankara, Turkey, where he is currently a professor. His research interests mainly include parallel computing, parallel scientific computing and its combinatorial aspects, parallel computer graphics applications, parallel data mining, graph and hypergraph theoretic models for load balancing, high-performance information retrieval systems, parallel and distributed databases, and grid computing. He has (co)authored about 60 . For more information on this or any other computing topic, please visit our Digital Library at www.computer.org/publications/dlib.

O O O B O O O B O O B O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Recommender System for Sport Videos Based on User Audiovisual Consumption Abstract-This paper describes a recommender system for sport videos, transmitted over the Internet and/or broadcast, in the context of large-scale events, which has been tested for the Olympic Games. The recommender is based on audiovisual consumption and does not depend on the number of users, running only on the client side. This avoids the concurrence, computation and privacy problems of central server approaches in scenarios with a large number of users, such as the Olympic Games.The system has been designed to take advantage of the information available in the videos, which is used along with the implicit information of the user and the modeling of his/her audiovisual content consumption. The system is thus transparent to the user, who does not need to take any specific action.Another important characteristic is that the system can produce recommendations for both live and recorded events.Testing has showed advantages compared to previous systems, as will be shown in the results.Index Terms-Audiovisual consumption, hidden Markov model (HMM), recommender system, sport videos. The most usual techniques for the generation of video recommendations are based on taking each video as a general item, without analyzing content characteristics. Following this approach, social techniques such as collaborative filtering [4] can be applied [5] . They are efficient and easy to apply. Another common technique is the use of standardized ontologies [6] , [7] which allows for the calculation of similarities using metrics as the Pearson correlation coefficient or the cosine measurement [8] .. However, the proposed video recommender system has to be adapted to a scenario with the aforementioned concurrence restrictions. These restrictions require that the recommender system runs only on the client side, therefore collaborative filtering or other social techniques cannot be used.. In this context, only content-based techniques can be used. A new algorithm based on the Winnow proposal [9] , which has been successfully used in other applications [10] , has been designed and adapted to be used in sport video applications. Xiao Wu et al. [11] also propose an interesting content-based Web video recommender.. Another important challenge of this research is the modeling of audiovisual consumption to infer the user's taste. This modeling has to be adapted to both Web and TV scenarios to allow its introduction in hybrid Web-TV systems. The most recent research in the field of audiovisual recommendation systems with implicit information has been developed by Hyoseop et al. [12] and Zhiwen et al. [13] . They propose systems that directly relate the consumption to the interest in a linear way, according to expression (1) : (1) is the rating inferred from the behavior of the user. The first term of the sum accounts for the audiovisual consumption:. is the user consumption period of an item, and is the total duration of the item. The other two terms of the sum, related to storage and bookmarks, are considered for recorded contents, not relevant in our scenario. Therefore, the final rating is only a proportional interpolation of the consumption time.. In this paper, the consumption modeling definition is an evolution of these approaches, including specific aspects described in Section V, with the addition of new modeling stages based on statistical and artificial intelligence processes (Section VI). Other studies about implicit information modeling which have influenced our approach are those of Syeda-Mahmood and Ponceleon [14] . They start recording some events, shot by certain users, to allow the user to preview videos in a search context. Then, they model some observable parameters as play, pause, stop, fast-forward, rewind, etc., by means of an HMM, which yields a particular description about the situation of the user with respect to an item: "Curious", "Aimless browse", "Found something interesting", "Abandoned interest", "Looking for something", "Resumed interest", "Found what I wanted", "Undetermined", "None of the above", "Did not find what I wanted".. Finally, Zimmerman et al. [15] propose a scenario closer to real-time recommendation. They use a rigid model to decide whether a user will consume a piece of content, according to his/her past behavior. They make a binary decision relying on the Bayesian classifier defined by Pazzani in 1998 [16] (and first proposed by Duda and Hart [17] ), and decision trees based on Quinlan studies [18] . To design the system, some rules are defined based on content metadata and user profile information.. In conclusion, these studies can be summarized into three different methods, which could be applicable to our scenario. Their characteristics are summarized in Table I and will be compared with our approach in Section VIII. user, and each has a certain magnitude associated, which lies within a discrete scale from 1 to 5. For example, for the user , the attribute Sport may have two possible values, football and basketball; if the first one is the favorite sport of , it will have a magnitude of 5; if the user likes basketball, but less than football, it will have a smaller value, such as 3 or 4. AVC measurement is applied to three of the four attributes, since for one of them, the Round attribute, it does not make sense. The AVC value for Round is constant for all users, because all users prefer (if this attribute is important for them) final rounds to first ones. It is important to distinguish both results, AR and AVC. For example, imagine two users who are only influenced by the Sport attribute: one of them only likes football and the other only likes basketball; AR value will be the same for both users, 5, whereas AVC value will be the following: 5 for football and 1 for basketball for the first user; 5 for basketball and 1 for football, for the second user.. The final predicted rating that the recommender system infers for one video associated to a specific user comes from the combination of both partial results, and , and will be explained in Section VII.. IV. SYSTEM ARCHITECTURE Three main stages can be observed in the system architecture. The end-to-end process starts with the acquisition of the user's audiovisual consumption data, and ends with the final recommendation. Fig. 1 shows the complete process, which is divided into three stages. In the consumption modeling stage, explained in Section V, the data from the user interface is acquired and a measurement of the local interest for the consumed item (a piece of video content) is generated. The measurement is then converted from the consumption domain to the attribute domain. In other words, the information provided by the last user consumption about the influence of each of the attributes is obtained. At this point, the information is split into four branches. The second stage, explained in Section VI, consists of four parallel processes. These processes infer the global importance of each attribute by taking into account both the past information about the user and the last observed consumption events (an event is a single visualization of a video by a user). The last stage, explained in Section VII, uses the updated values of AR and AVC to generate the final recommendations to the user.. The architecture presents a similar structure for the two scenarios considered, live and recorded videos. The live scenario presents an extra module in the final recommendation stage. This module performs a filtering of the corpora of videos, and selects those videos which correspond to ongoing or shortly upcoming competitions. This module also extracts data from the XML files which are generated during the video streaming and feeds them to the system.

B I O B I O O O B I O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Channel Estimation for Opportunistic Spectrum Access: Uniform and Random Sensing Abstract-The knowledge of channel statistics can be very helpful in making sound opportunistic spectrum access decisions. It is therefore desirable to be able to efficiently and accurately estimate channel statistics. In this paper, we study the problem of optimally placing sensing/sampling times over a time window so as to get the best estimate of the parameters of an on-off renewal channel. We are particularly interested in a sparse sensing regime with a small number of samples relative to the time window size. Using Fisher information as a measure, we analytically derive the best and worst sensing sequences under a sparsity condition. We also present a way to derive the best/worst sequences without this condition using a dynamic programming approach. In both cases the worst turns out to be the uniform sensing sequence, where sensing times are evenly spaced within the window. Interestingly the best sequence is also uniform but with a much smaller sensing interval that requires a priori knowledge of the channel parameters. With these results we argue that without a priori knowledge, a robust sensing strategy should be a randomized strategy. We then compare different random schemes using a family of distributions generated by the circular ensemble, and propose an adaptive sensing scheme to effectively track time-varying channel parameters. We further discuss the applicability of compressive sensing in the context of this problem. R ECENT advances in software-defined radio and cognitive radio [1] have given wireless devices greater ability and opportunity to dynamically access spectrum, thereby potentially significantly improving spectrum efficiency and user performance [2] , [3] . A key enabling ingredient is thus high-quality channel sensing that allows the user to obtain accurate real-time information on the condition of wireless channels. Spectrum sensing is often studied in two contexts: at the physical layer and at the MAC layer. Physical layer spectrum sensing typically focuses on the detection of instantaneous primary user signals. Methods like matched filter detection, energy detection, and feature detection have been proposed for cognitive radios [4] . MAC layer spectrum sensing [5] , [6] is more of a resource allocation issue, where we are concerned with the scheduling problem of when to sense the channel and the estimation problem of extracting statistical properties of the random variation in the channel, assuming that when we decide to sense the physical layer can provide sufficiently accurate results on instantaneous channel availability.. In this paper, we focus on the scheduling of channel sensing and study the effect different scheduling algorithms have on the accuracy of the resulting estimate we obtain on channel parameters. In particular, we are interested in a sparse sensing/sampling regime where we can use only a limited number of measurements over a given period of time. The goal is to decide how these limited number of measurements should be scheduled so as to minimize the estimation error within the maximum likelihood (ML) estimation framework. Throughout the paper the terms sensing and sampling will be used interchangeably.. MAC layer channel estimation has been studied in recent years. Below we review those most relevant to the present paper. In [5] , an ML estimator was introduced for renewal channels using a uniform sampling scheme where samples of the channel are taken at regular time intervals. A more accurate, but also much more computationally costly Bayesian estimator was introduced in [7] , again based on uniform sensing. Long et al. [8] analyzed the relationship between estimation accuracy, number of samples taken, and the channel state transition probabilities by using the sampling and estimation framework of [5] and focusing on Markovian channels. Park et al. [9] proposed a Hidden Markov Model (HMM)-based channel status predictor using reinforcement learning techniques. This predictor predicts next channel state based on past information obtained through uniformly sampling the channel. Fuqaha et al. [10] presented a channel estimation technique based on wavelet transform followed by filtering. This method relies on dense sampling of the channel.. In most of the above cited work, the focus is on the estimation problem given (sufficiently dense) uniform sampling of the channel, i.e., with equal time periods between successive samples. This scheme will be referred to as uniform sensing (US) in the remainder of this paper. By contrast, sampling schemes where time intervals between successive samples are drawn from a certain probability distribution will be referred to as random sensing (RS) throughout the paper. We observe that due to constraints on time, energy, memory, and other resources, a user may wish to perform channel sensing at much lower frequencies while still hoping for good estimates. This could be relevant for instance in cases where a user wants to track the channel condition in between active data communication, or where a user needs to track a large number of different channels. It is this sparse sampling scenario that we will focus on in this study, and the goal is to judiciously schedule these limited number of measurements.. Our main contributions are summarized as follows:. . We demonstrate that when sampling is done sparsely, random sensing significantly outperforms uniform sensing. . In the special case of exponentially distributed on/ off durations, we obtain the best and worst possible sampling schemes measured by the Fisher information. We show that uniform sensing is the worst; any deviation from it improves the estimation accuracy. Interestingly, the best sensing sequence is also uniform, but with a much smaller sampling interval that requires the knowledge of the underlying channel parameters. For this reason, this sequence is unimplementable but does shed light on the nature of the problem. . We show that under the same channel statistics and the same average sampling interval (or frequency), a random sensing scheme affects the estimation accuracy through the higher order central moments of the sampling intervals, and use the circular ensemble to study a family of distributions. . We present an adaptive random sensing scheme that can very effectively track time-varying channel parameters, and is shown to outperform its counterpart using uniform sensing. The remainder of this paper is organized as follows: Section 2 presents the channel models and Section 3 gives the detail of the ML estimator. Then in Section 4 we present how the sampling scheme affects the estimation performance; the best and worst sensing sequences with and without a sparse sampling condition are obtained. In Section 5, we use the circular ensemble to examine different random sampling schemes. Section 6 presents an adaptive random sensing scheme, and Section 7 concludes the paper. In this paper, we studied sensing schemes for a channel estimation problem under a sparsity condition. Using Fisher information as a performance measure, we derived the best and worst sensing sequences both with and without the sparsity condition. We then examined the performance of random sensing schemes, by comparing a family of distributions generated by the circular ensemble. Using these insights, an adaptive random sensing scheme was proposed to effectively track time-varying channel parameters.. Recent advances in compressive sensing theory [28] , [29] , [30] allow one to represent compressible/sparse signals with significantly fewer samples than required by the Nyquist sampling theorem. It is thus tempting to examine whether this technique brings any advantage for our channel estimation problem. The idea is to randomly sample the channel state, use compressive sensing techniques to reconstruct the entire sequence of channel state evolution, and then use the ML estimator to determine the channel parameter. So far we have not found this method to work well. The main obstacle lies in finding a good basis matrix that can both sparsify the channel signal vector and at the same time be sufficiently incoherent with the measurement matrix. A similar difficulty was noted in [31] in trying to use compressive sensing for a data gathering problem. This remains an interesting direction of future work.

B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Polygonal and polyhedral contour reconstruction in computed tomography Abstract-This paper is about three-dimensional (3-D) reconstruction of a binary image from its X-ray tomographic data. We study the special case of a compact uniform polyhedron totally included in a uniform background and directly perform the polyhedral surface estimation. We formulate this problem as a nonlinear inverse problem using the Bayesian framework. Vertice estimation is done without using a voxel approximation of the 3-D image. It is based on the construction and optimization of a regularized criterion that accounts for surface smoothness. We investigate original deterministic local algorithms, based on the exact computation of the line projections, their update, and their derivatives with respect to the vertice coordinates. Results are first derived in the two-dimensional (2-D) case, which consists of reconstructing a 2-D object of deformable polygonal contour from its tomographic data. Then, we investigate the 3-D extension that requires technical adaptations. Simulation results illustrate the performance of polygonal and polyhedral reconstruction algorithms in terms of quality and computation time.Index Terms-Block relaxation algorithms, closed contour and surface reconstruction, deformable template, global and local iterative descent algorithms, polygonal and polyhedral modeling, two-dimensional and three-dimensional binary image reconstruction from projections, X-ray tomography. We have studied general schemes for polygonal and polyhedral shape reconstruction from X-ray projection data. The proposed schemes directly estimate the position of the shape vertices using the MAP estimator. Minimization of posterior energy has been performed using deterministic first-order descent techniques. Among them, we have shown the interest of gradient techniques on the whole parameter vector and block relaxation techniques, where a block corresponds to a single vertice. Even though those algorithms are classic in numerical analysis, their use happens to be nontrivial for compact object reconstruction for two reasons: the nonconvexity of the minimization domain, and the direct computation of . The basis of our contribution is the direct computation, update, and differentiation of the projections with no use of a voxel approximation. These algorithms yield an efficient calculation method for , this quantity being required for the descent algorithms. In the 3-D case, the quality of reconstruction is similar for both techniques, but the computation burden is far higher for block relaxation. Therefore, we have selected gradient descent on . The self intersection test simply consists of the calculation of and its comparison with its finite difference approximation. Future work will primarily deal with the application of the polyhedron reconstruction method on real data in the field of nondestructive evaluation. On algorithmic parts, we think of using 2-D binary polygonal image sampling based on Arak fields [31] to perform stochastic optimization of in the polygonal case. Improvement of polyhedron reconstruction will aim to use nondifferentiable gradient techniques for the deterministic minimization of criterion [39] . Initial conditions can also be improved by the means of multiresolution techniques [34] , [40] . They consist of successive reconstructions of polyhedra with an increasing number of vertices. First estimates are very quick and afford initialization of the latter optimization stages. As a conclusion, the following theorem states the necessary and sufficient conditions for the differentiation of a triangle projection. The expression of the derivatives can be straightforwardly deduced from (19) - (23 We consider a tetrahedron and its projection of angle . We study the computation of and its derivability w.r.t. a single vertice, say . As for the 2-D projection of a triangle, function is piecewise affine and continuous on , provided that no face of is parallel to the direction of projection. 3 The support of is then equal to the convex hull of the geometric projections , , , and of vertices onto the detector plane (see Fig. 14) . The support is then either triangular or quadrangular, depending on the relative positions of . We now handle the calculation and differentiation of in both cases. We provide an analytic expression of and the necessary results to compute its derivatives.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I B I B I I I I I B I I I B I I I I I I I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Minimum d-blockers and d-transversals in graphs We consider a set V of elements and an optimization problem on V : the search for a maximum (or minimum) cardinality subset of V verifying a given property P. A d-transversal is a subset of V which intersects any optimum solution in at least d elements while a d-blocker is a subset of V whose removal deteriorates the value of an optimum solution by at least d. We present some general characteristics of these problems, we review some situations which have been studied (matchings, s − t paths and s − t cuts in graphs) and we study d-transversals and d-blockers for new problems as stable sets or vertex covers in bipartite graphs. In many instances occurring in practical situations or in a game theoretic context one faces the problem of preventing some move of an opponent with a minimum effort. Such problems have been studied from a theoretical point of view and, in very special cases where a graph representation may be used, it leads to interesting combinatorial problems (see for instance [16] , [15] ). The purpose of this paper is to discuss some of these problems and to present a few variations with some remarks on their complexity and on open questions which would lead to further developments. In the next section we shall provide a rather general formulation of the concepts of d-blockers and d-transversals with a brief review of earlier results in this direction. In Section 3, we present a general formulation of both problems as bi-level optimization programs. Then the following sections will be devoted to various special cases of applications of these concepts: combinatorial optimization problems in matroids, s-t paths and s-t cuts. In Section 7 we will consider the case of maximum stable sets in split graphs and in bipartite graphs. Finally Section 8 will give concluding remarks and suggestions for other research avenues. For all problems discussed here the determination of the optimal elements of C V was polynomial. It appears that the complexity of finding minimum d-transversals and d-blockers and the complexity of finding an optimal element of C V are not directly related. We have seen that in the case of line graphs of bipartite graphs the first problem is N P −hard while the second one is polynomial. Conversely, there are problems for which finding minimum d-transversals or d-blockers is polynomial while finding an optimal element of C V is N P −hard; such a situation occurs for instance when we want to find a d-transversal, d ≥ 1, in case where C V is the family of all longest elementary s − t paths, s has degree one and s is an endpoint of an induced path of length at least d. It could be interesting to examine other similar situations, i.e., where finding an optimal element of C V is N P −hard. In this paper, after having formalized the concepts of d-transversals and d-blockers we have solved the case where C V is the collection of all maximum stable sets in bipartite graphs. We have also solved the case of d-blockers in split graphs. It would be interesting to examine the case of other classes of graphs for which the complexity status is still open. One could consider other families C V which may lead to interesting results and to further research directions, like the weighted cases of the above problems.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A Robust Method for Handling Low Density Regions in Hybrid Simulations for Collisionless Plasmas A robust method to handle vacuum and near vacuum regions in hybrid simulations for space and astrophysical plasmas is presented. The conventional hybrid simulation model dealing with kinetic ions and a massless charge-neutralizing electron fluid is known to be susceptible to numerical instability due to divergence of the whistler-mode wave dispersion, as well as division-by-density operation in regions of low density. Consequently, a pure vacuum region is not allowed to exist in the simulation domain unless some ad hoc technique is used. To resolve this difficulty, an alternative way to introduce finite electron inertia effect is proposed. Contrary to the conventional method, the proposed one introduces a correction to the electric field rather than the magnetic field. It is shown that the generalized Ohm's law correctly reduces to Laplace's equation in a vacuum which therefore does not involve any numerical problems. In addition, a variable ion-to-electron mass ratio is introduced to reduce the phase velocity of high frequency whistler waves at low density regions so that the stability condition is always satisfied. It is demonstrated that the proposed model is able to handle near vacuum regions generated as a result of nonlinear self-consistent development of the system, as well as pure vacuum regions set up at the initial condition, without losing the advantages of the standard hybrid code. Numerical simulations have been an essential tool to investigate complicated nonlinear phenomena occurring in space and astrophysical plasmas. Although the conventional magnetohydrodynamics (MHD) proves itself useful to describe macroscopic plasma dynamics even in the collisionless regime in which the mean free path for Coulomb collisions is comparable to or larger than the system size, it does not necessarily means that one can completely ignore important kinetic physics. For example, it is well recognized that one must take into account kinetic effect to understand magnetic reconnection, which has been one of the key processes in magnetospheric physics affecting plasma transport, driving global convection, and perhaps triggering substorms. It is now becoming more and more popular to consider that magnetic reconnection plays a key role in astrophysical environments as well. Another example in which kinetic effect is central is the problem of particle acceleration in collisionless shocks. It requires seamless treatment of both microscopic and macroscopic physics because small-scale phenomena primarily determine the acceleration of low energy particles (or "injection"), while the transport of higher energy particles is predominantly governed by characteristics of MHD turbulence. Kinetic numerical simulations that can simultaneously deal with both macroscopic and microscopic dynamics of the collisionless plasma are indeed essential to investigate these important issues. Among those proposed so far, the best numerical technique for this purpose is probably the hybrid simulation, in which ions are treated kinetically whereas electrons are assumed to be a massless charge-neutralizing fluid (Winske et al., 2001; Lipatov, 2002) .. The concept of the hybrid simulation is indeed promising in that it enables us to access the ion dynamics, while seemingly less important but more computationally demanding electron physics has been factored out. It has been widely used to study elementary processes such as plasma instabilities, magnetic reconnection, collisionless fast and slow shocks (e.g., Leroy et al., 1982; Winske & Leroy, 1984; Terasawa et al., 1986; Nakamura et al., 1998; Higashimori & Hoshino, 2012) . With rapidly increasing computational resources, one may now be able to use a simulation box which is large enough to include the global scale as well. Recently, attempts have been made to model the interaction between the solar wind and relatively small unmagnetized and magnetized solar system bodies by using global hybrid simulations (e.g., Terada et al., 2002; Kallio & Janhunen, 2003; Trávníček et al., 2007; Holmström et al., 2012; Dyadechkin et al., 2013) . On the other hand, it has been well known that hybrid simulations are in practice susceptible to numerical instability. Despite the long history of this technique, to the authors knowledge, any fundamental solutions to this problem has not been given. It is indeed a serious obstacle that hinders application to many important and interesting problems in space and astrophysical plasma physics. The primary purpose of the present paper is to provide a practical solution to the problem of numerical stability in the hybrid simulation. As we will see below, this can be realized by introducing a new way to include finite electron inertia effect.. It is well known that the Alfvén wave at short wavelength comparable to ion inertia length has dispersion due to the decoupling between ion and electron dynamics. There thus appears the whistler mode whose frequency diverges as ω ∝ k 2 . This means that the maximum phase velocity in the system increases rapidly without bound, implying numerical difficulty. This is probably a part of the reasons for the numerical instability in hybrid simulations. It is thus easy to expect that inclusion of finite electron inertia can help stabilizing the simulation because the maximum phase velocity in this case is limited by roughly the electron Alfvén speed. Even with finite electron inertia, however, a numerical problem arises in regions of low density. This is obviously due to the division-by-density operation needed to calculate the electric field from ion moment quantities, which makes it impossible to handle such (near) vacuum regions. In practice, numerical difficulty arises even long before this limit is reached because the Alfvén speed increases as the density decreases, imposing a severe restriction on the simulation time step.. The method we propose in the present paper essentially resolves all these numerical difficulties. Our strategy is also to introduce finite electron inertia effect to limit the maximum phase velocity in the system. We argue that the way in which the electron inertia is introduced is a key to solve the problem. An electron inertia correction term has conventionally been introduced to the magnetic field and its electric field counterpart is often neglected (e.g., Kuznetsova et al., 1998; Shay et al., 1998; Nakamura et al., 2008) . By modifying the procedure so that the correction is introduced directly to the electric field, we show that the division-by-density operation is almost eliminated from the simulation procedure. In addition to this, to reduce the maximum wave phase velocity in a low density region, the ion-to-electron mass ratio is considered to be a variable quantity. That is, the mass ratio is reduced locally so that the CFL (Courant-Friedrichs-Lewy) condition is automatically satisfied. We demonstrate that the proposed model implemented in a one-dimensional (1D) hybrid simulation code can successfully follow nonlinear evolution of the system even when extremely low density regions appear as a result of strong instabilities. Furthermore, we also show that the code is able to handle pure vacuum regions, as well as the interface between vacuum and finite density plasma regions. These features suggest that the present model is indeed very robust and will help stabilizing simulations applied to many important problems in space and astrophysical plasmas.. The present paper is organized as follows. First, we present a simulation model in section 2, in which a new way to introduce finite electron inertia is discussed. Numerical implementation is explained in section 3. Section 4 shows simulation results of several test problems. Finally, summary and conclusions are given in section 5. In the present paper, we have introduced a new equation to determine the electric field for the hybrid simulations for collisionless plasmas. The equation takes into account finite electron inertia effect and reduces to Laplace's equation in the limit of low density. This is in clear contrast to the methods proposed so far that consider a correction only to the magnetic field. This difference resolves the fundamental difficulty inherent in the conventional hybrid simulation model, i.e, the impossibility of handling vacuum regions due to the existence of the division-by-density operation in the simulation procedure.. The present method improves numerical stability even for a region of finite density. This is because the inclusion of finite electron inertia imposes a limit of the maximum phase velocity of the system. In addition, it is sometimes needed for numerical stability to introduce a variable mass ratio technique, which ensures the stability by reducing the ion-to-electron mass ratio in regions where the maximum wave phase velocity violates the CFL stability condition. One may think that the electron inertia effect is merely a numerical stabilization factor rather than physics. This will hold in most of situations where the hybrid simulation applies and the electron inertia effect is not expected to be important, or not of primary interest. The proposed method will thus be useful for application of the hybrid code to problems where scale length comparable or longer than ion inertia length is essential, while appearance of low density regions as a result of self-consistent evolution of the system is unavoidable. It is worth mentioning that the method does not deteriorate the advantages of the standard hybrid code. In addition, we think that most of discussion presented in this paper will also apply to the Hall-MHD code as well. Finally, we note that although the current method introduces finite electron inertia, its application to problems where electron scale physics plays a role must be done with care, because the assumption of a fluid electron is not always appropriate for phenomena with scale length on the order of the electron inertia length encountered in space and astrophysical plasmas, unless electrons are sufficiently cold. where q s , f s (v) is charge and distribution function of particle species s, and the sum is taken over all particle species. Now we assume that the charged particles are collisionless and traveling under the action of electromagnetic fields. Then, the distribution function f s obeys the Vlasov equation. where the acceleration is given by the Lorentz force. One may now rewrite the right-hand side of Eq. (A.1) as where V s and P s are the bulk velocity and pressure tensor, respectively. Therefore, the contribution of the pressure gradient is greater from electrons than ions, unless the ion-to-electron temperature ratio is unusually high. The contribution of the term proportional to V s V s is usually small, but for electrons, it may not always be ignored because the electron bulk velocity can become as high as the electron Alfvén speed. Consequently, for typical problems to which the hybrid simulation is applied, it is sufficient to take into account If we further assume the charge neutrality condition n i ≈ n e , we have ∇ · E ≈ 0, and ∇ · V e ≈ 0, and Eq. (14) results. Note that the latter condition comes from the fact that temporal and spatial derivatives of the electron density are small on the electron scale, ∂n e /∂t ≈ ∂n e /∂x ≈ 0.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O