Simple , Fast , and Practical Non-Blocking and Blocking Concurrent Queue Algorithms ? Maged M. Michael Michael L. Scott Department of Computer Science University of Rochester Rochester , NY 14627-0226 fmichael,scottg@cs.rochester.edu December 1995 Abstract Drawing ideas from previous authors , we present a new non-blocking concurrent queue algorithm and a new two-lock queue algorithm in which one enqueue and one dequeue can proceed concurrently . Both algorithms are simple , fast , and practical ; we were surprised not to find them in the literature . Experiments on a 12-node SGI Challenge multiprocessor indicate that the new non-blocking queue consistently outperforms the best known alternatives ; it is the clear algorithm of choice for machines that provide a universal atomic primitive -LRB- e.g. compare and swap or load linked/store conditional -RRB- . The two-lock concurrent queue outperforms a single lock when several processes are competing simultaneously for access ; it appears to be the algorithm of choice for busy queues on machines with non-universal atomic primitives -LRB- e.g. test and set -RRB- . Since much of the motivation for non-blocking algorithms is rooted in their immunity to large , unpredictable delays in process execution , we report experimental results both for systems with dedicated processors and for systems with several processes multiprogrammed on each processor . ? This work was supported in part by NSF grants nos . CDA ? 94 ? 01142 and CCR ? 93 ? 19445 , and by ONR research grant no . N00014 ? 92 ? J ? 1801 -LRB- in conjunction with the DARPA Research in Information Science and Technology?High Performance Computing , Software Science and Technology program , ARPA Order no. 8930 -RRB- . 1 Introduction Concurrent FIFO queues are widely used in parallel applications and operating systems . To ensure correctness , concurrent access to shared queues has to be synchronized . Generally , algorithms for concurrent data structures , including FIFO queues , fall into two categories : blocking and non-blocking . Blocking algorithms allow a slow or delayed process to prevent faster processes from completing operations on the shared data structure indefinitely . Non-blocking algorithms guarantee that if there are one or more active processes trying to perform operations on a shared data structure , an operations will complete within finite number of time steps . On asynchronous -LRB- especially multiprogrammed -RRB- multiprocessor systems , blocking algorithms suffer significant performance degradation when a process is halted or delayed at an inopportune moment . Possible sources of delay include processor scheduling preemption , page faults , and cache misses . Non-blocking algorithms are more robust in the face of these events . Many researchers have proposed lock-free algorithms for concurrent FIFO queues . Hwang and Briggs -LSB- 7 -RSB- , Sites -LSB- 17 -RSB- , and Stone -LSB- 20 -RSB- present lock-free algorithms based on compare and swap .1 These algorithms are incompletely specified ; they omit details such as the handling of empty or single-item queues , or concurrent enqueues and dequeues . Lamport -LSB- 9 -RSB- presents a wait-free algorithm that restricts concurrency to a single enqueuer and a single dequeuer .2 Gottlieb et al. -LSB- 3 -RSB- and Mellor-Crummey -LSB- 11 -RSB- present algorithms that are lock-free but not non-blocking : they do not use locking mechanisms , but they allow a slow process to delay faster processes indefinitely . Treiber -LSB- 21 -RSB- presents an algorithm that is non-blocking but inefficient : a dequeue operation takes time proportional to the number of the elements in the queue . Herlihy -LSB- 6 -RSB- ; Prakash , Lee , and Johnson -LSB- 15 -RSB- ; Turek , Shasha , and Prakash -LSB- 22 -RSB- ; and Barnes -LSB- 2 -RSB- propose general methodologies for generating non-blocking versions of sequential or concurrent lock-based algorithms . However , the resulting implementations are generally inefficient compared to specialized algorithms . Herlihy and Wing -LSB- 4 -RSB- and Valois -LSB- 23 -RSB- present algorithms based on infinite arrays . Valois 's algorithm also requires an unaligned compare and swap . Massalin and Pu -LSB- 10 -RSB- present lock-free algorithms based on a double compare and swap primitive that operates on two arbitrary memory locations simultaneously , and that seems to be available only on later members of the Motorola 68000 family of processors . Stone -LSB- 18 -RSB- presents a queue that is lock-free but non-linearizable3 and not non-blocking . It is nonlinearizable because a slow enqueuer may cause a faster process to enqueue an item and subsequently observe an empty queue , even though the enqueued item has never been dequeued . It is not non-blocking because a slow enqueue can delay dequeues by other processes indefinitely . Our experiments also revealed a race condition in which a certain interleaving of a slow dequeue with faster enqueues and dequeues by other process -LRB- es -RRB- can cause an enqueued item to be lost permanently . Stone also presents -LSB- 19 -RSB- a non-blocking queue based on a circular singly-linked list . The algorithm uses one anchor pointer to manage the queue instead of the usual head and tail . Our experiments revealed a race condition in which a slow dequeuer can cause an enqueued item to be lost permanently . Prakash , Lee , and Johnson -LSB- 14 , 16 -RSB- present a linearizable non-blocking algorithm that requires enqueuing and dequeuing processes to take a snapshot of the queue in order to determine its ? state ? prior to updating it . The algorithm achieves the non-blocking property by allowing faster processes to complete the operations of slower processes instead of waiting for them . 1Compare and swap , introduced on the IBM System 370 , takes as arguments the address of a shared memory location , an expected value , and a new value . If the shared location currently holds the expected value , it is assigned the new value atomically . A Boolean return value indicates whether the replacement occurred . 2A wait-free algorithm is both non-blocking and starvation free : it guarantees that every active process will make progress within a bounded number of time steps . 3An implementation of a data structure is linearizable if it can always give an external observer , observing only the abstract data structure operations , the illusion that each of these operations takes effect instantaneously at some point between its invocation and its response -LSB- 5 -RSB- . Valois -LSB- 23 , 24 -RSB- presents a list-based non-blocking algorithm that avoids the contention caused by the snapshots of Prakash et al. 's algorithm and allows more concurrency by keeping a dummy node at the head -LRB- dequeue end -RRB- of a singly-linked list , thus simplifying the special cases associated with empty and singleitem queues -LRB- a technique suggested by Sites -LSB- 17 -RSB- -RRB- . Unfortunately , the algorithm allows the tail pointer to lag behind the head pointer , thus preventing dequeuing processes from safely freeing or re-using dequeued nodes . If the tail pointer lags behind and a process frees a dequeued node , the linked list can be broken , so that subsequently enqueued items are lost . Since memory is a limited resource , prohibiting memory reuse is not an acceptable option . Valois therefore proposes a special mechanism to free and allocate memory . The mechanism associates a reference counter with each node . Each time a process creates a pointer to a node it increments the node 's reference counter atomically . When it does not intend to access a node that it has accessed before , it decrements the associated reference counter atomically . In addition to temporary links from process-local variables , each reference counter reflects the number of links in the data structure that point to the node in question . For a queue , these are the head and tail pointers and linked-list links . A node is freed only when no pointers in the data structure or temporary variables point to it . We discovered and corrected -LSB- 13 -RSB- race conditions in the memory management mechanism and the associated non-blocking queue algorithm . Even so , the memory management mechanism and the queue that employs it are impractical : no finite memory can guarantee to satisfy the memory requirements of the algorithm all the time . Problems occur if a process reads a pointer to a node -LRB- incrementing the reference counter -RRB- and is then delayed . While it is not running , other processes can enqueue and dequeue an arbitrary number of additional nodes . Because of the pointer held by the delayed process , neither the node referenced by that pointer nor any of its successors can be freed . It is therefore possible to run out of memory even if the number of items in the queue is bounded by a constant . In experiments with a queue of maximum length 12 items , we ran out of memory several times during runs of ten million enqueues and dequeues , using a free list initialized with 64,000 nodes . Most of the algorithms mentioned above are based on compare and swap , and must therefore deal with the ABA problem : if a process reads a value A in a shared location , computes a new value , and then attempts a compare and swap operation , the compare and swap may succeed when it should not , if between the read and the compare and swap some other process -LRB- es -RRB- change the A to a B and then back to an A again . The most common solution is to associate a modification counter with a pointer , to always access the counter with the pointer in any read-modify-compare and swap sequence , and to increment it in each successful compare and swap . This solution does not guarantee that the ABA problem will not occur , but it makes it extremely unlikely . To implement this solution , one must either employ a double-word compare and swap , or else use array indices instead of pointers , so that they may share a single word with a counter . Valois 's reference counting technique guarantees preventing the ABA problem without the need for modification counters or the double-word compare and swap . MellorCrummey 's lock-free queue -LSB- 11 -RSB- requires no special precautions to avoid the ABA problem because it uses compare and swap in a fetch and store-modify-compare and swap sequence rather than the usual read-modify-compare and swap sequence . However , this same feature makes the algorithm blocking . In section 2 we present two new concurrent FIFO queue algorithms inspired by ideas in the work described above . Both of the algorithms are simple and practical . One is non-blocking ; the other uses a pair of locks . Correctness of these algorithms is discussed in section 3 . We present experimental results in section 4 . Using a 12-node SGI Challenge multiprocessor , we compare the new algorithms to a straightforward single-lock queue , Mellor-Crummey 's blocking algorithm -LSB- 11 -RSB- , and the non-blocking algorithms of Prakash et al. -LSB- 16 -RSB- and Valois -LSB- 24 -RSB- , with both dedicated and multiprogrammed workloads . The results confirm the value of non-blocking algorithms on multiprogrammed systems . They also show consistently superior performance on the part of the new lock-free algorithm , both with and without multiprogramming . The new two-lock algorithm can not compete with the non-blocking alternatives on a multiprogrammed system , but outperforms a single lock when several processes compete for access simultaneously . Section 5 summarizes our conclusions . 2 Algorithms Figure 1 presents commented pseudo-code for the non-blocking queue data structure and operations . The algorithm implements the queue as a singly-linked list with Head and Tail pointers . As in Valois 's algorithm , Head always points to a dummy node , which is the first node in the list . Tail points to either the last or second to last node in the list . The algorithm uses compare and swap , with modification counters to avoid the ABA problem . To allow dequeuing processes to free dequeued nodes , the dequeue operation ensures that Tail does not point to the dequeued node nor to any of its predecessors . This means that dequeued nodes may safely be re-used . To obtain consistent values of various pointers we rely on sequences of reads that re-check earlier values to be sure they have n't changed . These sequences of reads are similar to , but simpler than , the snapshots of Prakash et al. -LRB- we need to check only one shared variable rather than two -RRB- . A similar technique can be used to prevent the race condition in Stone 's blocking algorithm . We use Treiber 's simple and efficient non-blocking stack algorithm -LSB- 21 -RSB- to implement a non-blocking free list . Figure 2 presents commented pseudo-code for the two-lock queue data structure and operations . The algorithm employs separate Head and Tail locks , to allow complete concurrency between enqueues and dequeues . As in the non-blocking queue , we keep a dummy node at the beginning of the list . Because of the dummy node , enqueuers never have to access Head , and dequeuers never have to access Tail , thus avoiding potential deadlock problems that arise from processes trying to acquire the locks in different orders . 3 Correctness 3.1 Safety We show that the presented algorithms are safe by showing that they satisfy the following properties : 1 . The linked list is always connected . 2 . Nodes are only inserted after the last node in the linked list . 3 . Nodes are only deleted from the beginning of the linked list . 4 . Head always points to the first node in the linked list . 5 . Tail always point to a node in the linked list . Initially , all these properties hold . By induction , we show that they continue to hold , assuming that the ABA problem never occurs . 1 . The linked list is always connected because once a node is inserted , its next pointer is not set to NULL before it is freed , and no node is freed until it is deleted from the beginning of the list -LRB- property 3 -RRB- . 2 . In the lock-free algorithm , nodes are only inserted at the end of the linked list because they are linked through the Tail pointer , which always points to a node in the linked-list -LRB- property 5 -RRB- , and an inserted node is linked only to a node that has a NULL next pointer , and the only such node in the linked list is the last one -LRB- property 1 -RRB- . structure pointer t fptr : pointer to node t , count : unsigned integerg structure node t fvalue : data type , next : pointer tg structure queue t fHead : pointer t , Tail : pointer tg initialize -LRB- Q : pointer to queue t -RRB- node = new node -LRB- -RRB- # Allocate a free node node ? > next.ptr = NULL # Make it the only node in the linked list Q ? > Head = Q ? > Tail = node # Both Head and Tail point to it enqueue -LRB- Q : pointer to queue t , value : data type -RRB- E1 : node = new node -LRB- -RRB- # Allocate a new node from the free list E2 : node ? > value = value # Copy enqueued value into node E3 : node ? > next.ptr = NULL # Set next pointer of node to NULL E4 : repeat # Keep trying until Enqueue is done E5 : tail = Q ? > Tail # Read Tail.ptr and Tail.count together E6 : next = tail.ptr ? > next # Read next ptr and count fields together E7 : if tail == Q ? > Tail # Are tail and next consistent ? E8 : if next.ptr == NULL # Was Tail pointing to the last node ? E9 : if CAS -LRB- & tail.ptr ? > next , next , < node , next.count +1 > -RRB- # Try to link node at the end of the linked list E10 : break # Enqueue is done . Exit loop E11 : else # Tail was not pointing to the last node E12 : CAS -LRB- & Q ? > Tail , tail , < next.ptr , tail.count +1 > -RRB- # Try to swing Tail to the next node E13 : CAS -LRB- & Q ? > Tail , tail , < node , tail.count +1 > -RRB- # Enqueue is done . Try to swing Tail to the inserted node dequeue -LRB- Q : pointer to queue t , pvalue : pointer to data type -RRB- : boolean D1 : repeat # Keep trying until Dequeue is done D2 : head = Q ? > Head # Read Head D3 : tail = Q ? > Tail # Read Tail D4 : next = head ? > next # Read Head.ptr ? > next D5 : if head == Q ? > Head # Are head , tail , and next consistent ? D6 : if head.ptr == tail.ptr # Is queue empty or Tail falling behind ? D7 : if next.ptr == NULL # Is queue empty ? D8 : return FALSE # Queue is empty , could n't dequeue D9 : CAS -LRB- & Q ? > Tail , tail , < next.ptr , tail.count +1 > -RRB- # Tail is falling behind . Try to advance it D10 : else # No need to deal with Tail # Read value before CAS , otherwise another dequeue might free the next node D11 : * pvalue = next.ptr ? > value D12 : if CAS -LRB- & Q ? > Head , head , < next.ptr , head.count +1 > -RRB- # Try to swing Head to the next node D13 : break # Dequeue is done . Exit loop D14 : free -LRB- head.ptr -RRB- # It is safe now to free the old dummy node D15 : return TRUE # Queue was not empty , dequeue succeeded Figure 1 : Structure and operation of a non-blocking concurrent queue . structure node t fvalue : data type , next : pointer to node tg structure queue t fHead : pointer to node t , Tail : pointer to node t , H lock : lock type , T lock : lock typeg initialize -LRB- Q : pointer to queue t -RRB- node = new node -LRB- -RRB- # Allocate a free node node ? > next.ptr = NULL # Make it the only node in the linked list Q ? > Head = Q ? > Tail = node # Both Head and Tail point to it Q ? > H lock = Q ? > T lock = FREE # Locks are initially free enqueue -LRB- Q : pointer to queue t , value : data type -RRB- node = new node -LRB- -RRB- # Allocate a new node from the free list node ? > value = value # Copy enqueued value into node node ? > next.ptr = NULL # Set next pointer of node to NULL lock -LRB- & Q ? > T lock -RRB- # Acquire T lock in order to access Tail Q ? > Tail ? > next = node # Link node at the end of the linked list Q ? > Tail = node # Swing Tail to node unlock -LRB- & Q ? > T lock -RRB- # Release T lock dequeue -LRB- Q : pointer to queue t , pvalue : pointer to data type -RRB- : boolean lock -LRB- & Q ? > H lock -RRB- # Acquire H lock in order to access Head node = Q ? > Head # Read Head new head = node ? > next # Read next pointer if new head == NULL # Is queue empty ? unlock -LRB- & Q ? > H lock -RRB- # Release H lock before return return FALSE # Queue was empty * pvalue = new head ? > value # Queue not empty . Read value before release Q ? > Head = new head # Swing Head to next node unlock -LRB- & Q ? > H lock -RRB- # Release H lock free -LRB- node -RRB- # Free node return TRUE # Queue was not empty , dequeue succeeded Figure 2 : Structure and operation of a two-lock concurrent queue . In the lock-based algorithm nodes are only inserted at the end of the linked list because they are inserted after the node pointed to by Tail , and in this algorithm Tail always points to the last node in the linked list , unless it is protected by the tail lock . 3 . Nodes are deleted from the beginning of the list , because they are deleted only when they are pointed to by Head and Head always points to the first node in the list -LRB- property 4 -RRB- . 4 . Head always points to the first node in the list , because it only changes its value to the next node atomically -LRB- either using the head lock or using compare and swap -RRB- . When this happens the node it used to point to is considered deleted from the list . The new value of Head can not be NULL because if there is one node in the linked list the dequeue operation returns without deleting any nodes . 5 . Tail always points to a node in the linked list , because it never lags behind Head , so it can never point to a deleted node . Also , when Tail changes its value it always swings to the next node in the list and it never tries to change its value if the next pointer is NULL . 3.2 Linearizability The presented algorithms are linearizable because there is a specific point during each operation at which it is considered to ? take effect ? -LSB- 5 -RSB- . An enqueue takes effect when the allocated node is linked to the last node in the linked list . A dequeue takes effect when Head swings to the next node . And , as shown in the previous subsection -LRB- properties 1 , 4 , and 5 -RRB- , the queue variables always reflect the state of the queue ; they never enter a transient state in which the state of the queue can be mistaken -LRB- e.g. a non-empty queue appears to be empty -RRB- . 3.3 Liveness The Lock-Free Algorithm is Non-Blocking We show that the lock-free algorithm is non-blocking by showing that if there are non-delayed processes attempting to perform operations on the queue , an operation is guaranteed to complete within finite time . An enqueue operation loops only if the condition in line E7 fails , the condition in line E8 fails , or the compare and swap in line E9 fails . A dequeue operation loops only if the condition in line D5 fails , the condition in line D6 holds -LRB- and the queue is not empty -RRB- , or the compare and swap in line D12 fails . We show that the algorithm is non-blocking by showing that a process loops beyond a finite number of times only if another process completes an operation on the queue . ffl The condition in line E7 fails only if Tail is written by an intervening process after executing line E5 . Tail always points to the last or second to last node of the linked list , and when modified it follows the next pointer of the node it points to . Therefore , if the condition in line E7 fails more than once , then another process must have succeeded in completing an enqueue operation . ffl The condition in line E8 fails if Tail was pointing to the second to last node in the linked-list . After the compare and swap in line E12 , Tail must point to the last node in the list , unless a process has succeeded in enqueuing a new item . Therefore , if the condition in line E8 fails more than once , then another process must have succeeded in completing an enqueue operation . ffl The compare and swap in line E9 fails only if another process succeeded in enqueuing a new item to the queue . ffl The condition in line D5 and the compare and swap in line D12 fail only if Head has been written by another process . Head is written only when a process succeeds in dequeuing an item . ffl The condition in line D6 succeeds -LRB- while the queue is not empty -RRB- only if Tail points to the second to last node in the linked list -LRB- in this case it is also the first node -RRB- . After the compare and swap in line D9 , Tail must point to the last node in the list , unless a process succeeded in enqueuing a new item . Therefore , if the condition of line D6 succeeds more than once , then another process must have succeeded in completing an enqueue operation -LRB- and the same or another process succeeded in dequeuing an item -RRB- . The Two-Lock Algorithm is Livelock-Free The two-lock algorithm does not contain any loops . Therefore , if the mutual exclusion lock algorithm used for locking and unlocking the head and tail locks is livelock-free , then the presented algorithm is livelock-free too . There are many mutual exclusion algorithms that are livelock-free -LSB- 12 -RSB- . 4 Performance We use a 12-processor Silicon Graphics Challenge multiprocessor to compare the performance of the new algorithms to that of a single-lock algorithm , the algorithm of Prakash et al. -LSB- 16 -RSB- , Valois 's algorithm -LSB- 24 -RSB- -LRB- with corrections to the memory management mechanism -LSB- 13 -RSB- -RRB- , and Mellor-Crummey 's algorithm -LSB- 11 -RSB- . We include the algorithm of Prakash et al. because it appears to be the best of the known non-blocking alternatives . Mellor-Crummey 's algorithm represents non-lock-based but blocking alternatives ; it is simpler than the code of Prakash et al. , and could be expected to display lower constant overhead in the absence of unpredictable process delays , but is likely to degenerate on a multiprogrammed system . We include Valois 's algorithm to demonstrate that on multiprogrammed systems even a comparatively inefficient non-blocking algorithm can outperform blocking algorithms . For the two lock-based algorithms we use test-and-test and set locks with bounded exponential backoff -LSB- 12 , 1 -RSB- . We also use backoff where appropriate in the non-lock-based algorithms . Performance was not sensitive to the exact choice of backoff parameters in programs that do at least a modest amount of work between queue operations . We emulate both test and set and the atomic operations required by the other algorithms -LRB- compare and swap , fetch and increment , fetch and decrement , etc. -RRB- using the MIPS R4000 load linked and store conditional instructions . C code for the tested algorithms can be obtained from ftp : / / ftp.cs.rochester.edu/pub/ packages/sched conscious synch/concurrent queues . The algorithms were compiled at the highest optimization level , and were carefully hand-optimized . We tested each of the algorithms in hours-long executions on various numbers of processors . It was during this process that we discovered the race conditions mentioned in section 1 . All the experiments employ an initially-empty queue to which processes perform a series of enqueue and dequeue operations . Each process enqueues an item , does ? other work ? , dequeues an item , does ? other work ? , and repeats . With p processes , each process executes this loop b106 = pc or d106 = pe times , for a total of one million enqueues and dequeues . The ? other work ? consists of approximately 6 ? s of spinning in an empty loop ; it serves to make the experiments more realistic by preventing long runs of queue operations by the same process -LRB- which would display overly-optimistic performance due to an unrealistically low cache miss rate -RRB- . We subtracted the time required for one processor to complete the ? other work ? from the total time reported in the figures . Figure 3 shows net elapsed time in seconds for one million enqueue/dequeue pairs . Roughly speaking , this corresponds to the time in microseconds for one enqueue/dequeue pair . More precisely , for k processors , the graph shows the time one processor spends performing 106 = k enqueue/dequeue pairs , plus the amount by which the critical path of the other 106 -LRB- k ? 1 -RRB- = k pairs performed by other processors exceeds the time spent by the first processor in ? other work ? and loop overhead . For k = 1 , the second term is zero . As k increases , the first term shrinks toward zero , and the second term approaches the critical path length of the overall computation ; i.e. one million times the serial portion of an enqueue/dequeue pair . Exactly how much execution will overlap in different processors depends on the choice of algorithm , the number of processors k , and the length of the ? other work ? between queue operations . With only one processor , memory references in all but the first loop iteration hit in the cache , and completion times are very low . With two processors active , contention for head and tail pointers and queue elements causes a high fraction of references to miss in the cache , leading to substantially higher completion times . The queue operations of processor 2 , however , fit into the ? other work ? time of processor 1 , and vice versa , so we are effectively measuring the time for one processor to complete 5 ? 105 enqueue/dequeue pairs . At three processors , the cache miss rate is about the same as it was with two processors . Each processor only has to perform 106 = 3 enqueue/dequeue pairs , but some of the operations of the other processors no longer fit in the first processor 's ? other work ? time . Total elapsed time decreases , but by a fraction less than 1 = 3 . Toward the right-hand side of the graph , execution time rises for most algorithms as smaller and smaller amounts of per-processor ? other work ? and loop overhead are subtracted from a total time dominated by critical path length . In the single-lock and Mellor-Crummey curves , the increase is probably accelerated as high rates of contention increase the average cost of a cache miss . In Valois 's algorithm , the plotted time continues to decrease , as more and more of the memory management overhead moves out of the critical path and into the overlapped part of the computation . Figures 4 and 5 plot the same quantity as figure 3 , but for a system with 2 and 3 processes per processor , respectively . The operating system multiplexes the processor among processes with a scheduling quantum of 10 ms. As expected , the blocking algorithms fare much worse in the presence of multiprogramming , since an inopportune preemption can block the progress of every process in the system . Also as expected , the degree of performance degradation increases with the level of multiprogramming . In all three graphs , the new non-blocking queue outperforms all of the other alternatives when three or more processors are active . Even for one or two processors , its performance is good enough that we can comfortably recommend its use in all situations . The two-lock algorithm outperforms the one-lock algorithm when more than 5 processors are active on a dedicated system : it appears to be a reasonable choice for machines that are not multiprogrammed , and that lack a universal atomic primitive -LRB- compare and swap or load linked/store conditional -RRB- . 5 Conclusions Queues are ubiquitous in parallel programs , and their performance is a matter of major concern . We have presented a concurrent queue algorithm that is simple , non-blocking , practical , and fast . We were surprised not to find it in the literature . It seems to be the algorithm of choice for any queue-based application on a multiprocessor with universal atomic primitives -LRB- e.g. compare and swap or load linked/store conditional -RRB- . We have also presented a queue with separate head and tail pointer locks . Its structure is similar to that of the non-blocking queue , but it allows only one enqueue and one dequeue to proceed at a given time . Because it is based on locks , however , it will work on machines with such simple atomic primitives as test and set . We recommend it for heavily-utilized queues on such machines -LRB- For a queue that is usually accessed by only one or two processors , a single lock will run a little faster . -RRB- This work is part of a larger project that seeks to evaluate the tradeoffs among alternative mechanisms for atomic update of common data structures . Structures under consideration include stacks , queues , heaps , search trees , and hash tables . Mechanisms include single locks , data-structure-specific multilock algorithms , general-purpose and special-purpose non-blocking algorithms , and function shipping to a centralized manager -LRB- a valid technique for situations in which remote access latencies dominate computation time -RRB- . In related work -LSB- 8 , 25 , 26 -RSB- , we have been developing general-purpose synchronization mechanisms that cooperate with a scheduler to avoid inopportune preemption . Given that immunity to processes delays is a primary benefit of non-blocking parallel algorithms , we plan to compare these two approaches in the context of multiprogrammed systems . References -LSB- 1 -RSB- T. E. Anderson . The Performance of Spin Lock Alternatives for Shared-Memory Multiprocessors . IEEE Transactions on Parallel and Distributed Systems , 1 -LRB- 1 -RRB- :6 ? 16 , January 1990 . 1 2 3 4 5 6 7 8 9 10 11 12 5 10 15 20 25 Processors seconds Single lock MC lock-free Valois non-blocking new two-lock PLJ non-blocking new non-blocking Figure 3 : Net execution time for one million enqueue/dequeue pairs on a dedicated multiprocessor . 1 2 3 4 5 6 7 8 9 10 11 12 5 10 15 20 25 Processors seconds Figure 4 : Net execution time for one million enqueue/dequeue pairs on a multiprogrammed system with 2 processes per processor . 1 2 3 4 5 6 7 8 9 10 11 12 5 10 15 20 25 Processors seconds Figure 5 : Net execution time for one million enqueue/dequeue pairs on a multiprogrammed system with 3 processes per processor . -LSB- 2 -RSB- G. Barnes . A Method for Implementing Lock-Free Data Structures . In Proceedings of the Fifth Annual ACM Symposium on Parallel Algorithms and Architectures , Velen , Germany , June ? July 1993 . -LSB- 3 -RSB- A. Gottlieb , B. D. Lubachevsky , and L. Rudolph . Basic Techniques for the Efficient Coordination of Very Large Numbers of Cooperating Sequential Processors . ACM Transactions on Programming Languages and Systems , 5 -LRB- 2 -RRB- :164 ? 189 , April 1983 . -LSB- 4 -RSB- M. P. Herlihy and J. M. Wing . Axions for Concurrent Objects . In Proceedings of the 14th ACM Symposium on Principles of Programming Languages , pages 13 ? 26 , January 1987 . -LSB- 5 -RSB- M. P. Herlihy and J. M. Wing . Linearizability : A Correctness Condition for Concurrent Objects . ACM Transactions on Programming Languages and Systems , 12 -LRB- 3 -RRB- :463 ? 492 , July 1990 . -LSB- 6 -RSB- M. Herlihy . A Methodology for Implementing Highly Concurrent Data Objects . ACM Transactions on Programming Languages and Systems , 15 -LRB- 5 -RRB- :745 ? 770 , November 1993 . -LSB- 7 -RSB- K. Hwang and F. A. Briggs . Computer Architecture and Parallel Processing . McGraw-Hill , 1984 . -LSB- 8 -RSB- L. Kontothanassis and R. Wisniewski . Using Scheduler Information to Achieve Optimal Barrier Synchronization Performance . In Proceedings of the Fourth ACM Symposium on Principles and Practice of Parallel Programming , May 1993 . -LSB- 9 -RSB- L. Lamport . Specifying Concurrent Program Modules . ACM Transactions on Programming Languages and Systems , 5 -LRB- 2 -RRB- :190 ? 222 , April 1983 . -LSB- 10 -RSB- H. Massalin and C. Pu . A Lock-Free Multiprocessor OS Kernel . Technical Report CUCS-005-91 , Computer Science Department , Columbia University , 1991 . -LSB- 11 -RSB- J. M. Mellor-Crummey . Concurrent Queues : Practical Fetch-and-F Algorithms . TR 229 , Computer Science Department , University of Rochester , November 1987 . -LSB- 12 -RSB- J. M. Mellor-Crummey and M. L. Scott . Algorithms for Scalable Synchronization on Shared-Memory Multiprocessors . ACM Transactions on Computer Systems , 9 -LRB- 1 -RRB- :21 ? 65 , February 1991 . -LSB- 13 -RSB- M. M. Michael and M. L. Scott . Correction of a Memory Management Method for Lock-Free Data Structures . Technical Report 599 , Computer Science Department , University of Rochester , December 1995 . -LSB- 14 -RSB- S. Prakash , Y. H. Lee , and T. Johnson . A Non-Blocking Algorithm for Shared Queues Using Compare-and Swap . In Proceedings of the 1991 International Conference on Parallel Processing , pages II :68 ? 75 , 1991 . -LSB- 15 -RSB- S. Prakash , Y. H. Lee , and T. Johnson . Non-Blocking Algorithms for Concurrent Data Structures . Technical Report 91-002 , University of Florida , 1991 . -LSB- 16 -RSB- S. Prakash , Y. H. Lee , and T. Johnson . A Nonblocking Algorithm for Shared Queues Using Compareand-Swap . IEEE Transactions on Computers , 43 -LRB- 5 -RRB- :548 ? 559 , May 1994 . -LSB- 17 -RSB- R. Sites . Operating Systems and Computer Architecture . In H. Stone , editor , Introduction to Computer Architecture , 2nd edition , Chapter 12 , 1980 . Science Research Associates . -LSB- 18 -RSB- J. M. Stone . A Simple and Correct Shared-Queue Algorithm Using Compare-and-Swap . In Proceedings Supercomputing ' 90 , November 1990 . -LSB- 19 -RSB- J. M. Stone . A Non-Blocking Compare-and-Swap Algorithm for a Shared Circular Queue . In S. Tzafestas et al. , editors , Parallel and Distributed Computing in Engineering Systems , pages 147 ? 152 , 1992 . Elsevier Science Publishers . -LSB- 20 -RSB- H. S. Stone . High Performance Computer Architecture . Addison-Wesley , 1993 . -LSB- 21 -RSB- R. K. Treiber . Systems Programming : Coping with Parallelism . In RJ 5118 , IBM Almaden Research Center , April 1986 . -LSB- 22 -RSB- J. Turek , D. Shasha , and S. Prakash . Locking without Blocking : Making Lock Based Concurrent Data Structure Algorithms Nonblocking . In Proceedings of the 11th ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems , pages 212 ? 222 , 1992 . -LSB- 23 -RSB- J. D. Valois . Implementing Lock-Free Queues . In Seventh International Conference on Parallel and Distributed Computing Systems , Las Vegas , NV , October 1994 . -LSB- 24 -RSB- J. D. Valois . Lock-Free Data Structures . Ph. D. dissertation , Rensselaer Polytechnic Institute , May 1995 . -LSB- 25 -RSB- R. W. Wisniewski , L. Kontothanassis , and M. L. Scott . Scalable Spin Locks for Multiprogrammed Systems . In Proceedings of the Eighth International Parallel Processing Symposium , pages 583 ? 589 , Cancun , Mexico , April 1994 . Earlier but expanded version available as TR 454 , Computer Science Department , University of Rochester , April 1993 . -LSB- 26 -RSB- R. W. Wisniewski , L. I. Kontothanassis , and M. L. Scott . High Performance Synchronization Algorithms for Multiprogrammed Multiprocessors . In Proceedings of the Fifth ACM Symposium on Principles and Practice of Parallel Programming , Santa Barbara , CA , July 1995 .

O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O B I I O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

compare and swap
concurrent queue
multiprogramming


Discretization of Continuous-Valued Attributes and Instance-Based Learning Ting , Kai Ming Basser Department of Computer Science University of Sydney , NSW 2006 , Australia E-mail : kaiming@cs.su.oz.au Technical Report No. 491 October , 1994 Abstract Recent work on discretization of continuous-valued attributes in learning decision trees has produced some positive results . This paper adopts the idea of discretization of continuous-valued attributes and applies it to instance-based learning -LRB- Aha , 1990 ; Aha , Kibler & Albert , 1991 -RRB- . Our experiments have shown that instance-based learning -LRB- IBL -RRB- usually performs well in continuous-valued attribute domains and poorly in nominal attribute domains . Cost and Salzberg -LRB- 1993 -RRB- have devised the modified value-difference metric -LRB- MVDM -RRB- that raises the performance of IBL in nominal attribute domains . This paper explores a way in which continuous-valued attributes and nominal attributes can be treated cohesively in IBL . An algorithm which combines the discretization of continuous-valued attributes and IB1 -LRB- Aha , Kibler & Albert , 1991 -RRB- using the modified value-difference metric is introduced . The empirical results show that the proposed algorithm , IB1-MVDM * achieves a substantial improvement over C4 .5 -LRB- Quinlan , 1993 -RRB- , IB1 and IB1-MVDM in most of the domains tested . A performance comparison is also made with a naive Bayesian learner -LRB- Cestnik , 1990 -RRB- . 1 Introduction Virtually no work has been done in treating continuous-valued attributes and nominal attributes in a coherent manner in the framework of instancebased learning . The treatment of continuous-valued attributes in instancebased learning has been acknowledged to achieve satisfactory results . Recent work has been concentrating on improving the treatment of nominal attributes -LRB- Cost & Salzberg , 1993 -RRB- and encouraging results have been attained . As a result of this development , the performance of instance-based learning in nominal attribute domains seems to surpass those in continuousvalued attribute domains . It is the difference in performance in continuousvalued attribute domains and nominal attribute domains that motivates the work reported in this paper . Relative little research on the discretization of continuous-valued attributes has been reported in the machine learning research literature . Two recent works -LRB- Fayyad & Irani , 1993 ; Van de Merckt , 1993 -RRB- under the framework of decision trees have benefited the work reported here . The objective of this paper is to explore a way in which continuous-valued attributes and nominal attributes can be treated cohesively in instancebased learning . We begin the next section with a brief description of instance-based learning and a recent improvement in the treatment of nominal attributes . Section 3 reports in brief three current methods for discretizing of continuous-valued attributes . A description of the experimental design is given in the next section . Section 5 presents the results and is followed by discussion and future work . 2 Instance-Based Learning Instance-based learning -LRB- IBL -RRB- distinguishes itself from other types of learning that induce theories in the forms of decision trees , rules and neural networks . IBL simply stores the training instances and classifies a new instance by predicting that it and its nearest stored instance -LRB- according to some distance measures -RRB- have the same class -LRB- Stanfill & Waltz , 1986 ; Aha , Kibler & Albert , 1991 ; Cost & Salzberg , 1993 -RRB- . For a continuous-valued attribute , the value-difference between two values is simply an arithmetic difference . In a nominal attributes domain , the \ overlap '' metric is usually used , counting the number of attributes that differ . In order to handle domains that have both types of attributes , the overlap metric is also applied in the same Euclidean distance formulation used for continuous-valued attributes -LRB- e.g. the IB series algorithms by Aha , Kibler & Albert , 1991 -RRB- . Cost and Salzberg -LRB- 1993 -RRB- observe that IBL performs relatively poor in nominal attribute domains and identify that it is due to the use of the overlap metric . They use a modified version of the value-difference metric -LRB- MVDM -RRB- , first introduced by Stanfill & Waltz -LRB- 1986 -RRB- , and empirically show that MVDM is a better metric for nominal attribute domains . The value-difference distance between two possible values -LRB- v1 ; v2 -RRB- of a specific attribute is defined as follows -LRB- Cost & Salzberg , 1993 -RRB- : ffi -LRB- v1 ; v2 -RRB- = nX i = 1 C1 ; i C1 ? C2 ; i C2 where C1 ; i is the number of times v1 was classified into categories i , and C1 is the total number of times value v1 occurred . In the experiment reported in Section 5 , two IBL algorithms are used , i.e. IB1 -LRB- Aha , Kibler & Albert , 1991 -RRB- and IB1-MVDM , a modified version of IB1 that incorporates MVDM rather than the overlap metric for nominal attributes . Our independent experiment -LRB- see Table 1 -RRB- using IB1-MVDM in nominal attribute domains confirms Cost and Salzberg 's claim . Further experiment with continuous-valued domains and comparison with C4 .5 seem to reveal two phenomena : i. For nominal attribute domains , IB1-MVDM outperforms C4 .5 with a large improvement . ii . For continuous-valued attribute domains , IB1-MVDM/IB1 's performance is comparable to C4 .5 . This observation raises doubt as to whether the metric used thus far for continuous-valued attributes is a good metric . Instead of looking for another metric for continuous-valued attributes , a method is sought to convert continuous-valued attributes to nominal attributes at the outset . We postulate that if a proper method can be found , IB1-MVDM would outperform C4 .5 in both continuous-valued attribute and nominal attribute domains . Domain C4 .5 IB1-MVDM IB1 bcw 5.4 % 4.5 % diabetes 29.3 % 29.8 % waveform 30.8 % 36.5 % heart 25.9 % 24.7 % glass 32.3 % 30.0 % lymphography 22.1 % 16.4 % 21.7 % promoter 22.6 % 7.1 % 19.3 % soybean 8.1 % 6.0 % 8.5 % monks-2 35.0 % 25.5 % 29.6 % nettalk -LRB- phoneme -RRB- 19.7 % 16.2 % 26.1 % Table 1 : Error rates of three algorithms Another appeal of using IB1-MVDM and converting continuous-valued attributes to nominal attributes at the outset is that it avoids any problems arising from the combination of two different metrics . Because separate metrics are used for continuous-valued and nominal attributes , this may bring about poor results . Before we show the experiments and their results , let us examine the methods of discretization of continuous-valued attributes in the next section . 3 Discretization of Continuous-Valued At - tributes We describe three discretization methods used in decision trees in this section and use these methods for the experiments described in Section 4 . Fayyad and Irani -LRB- 1993 -RRB- devised a method for multi-value splitting for continuous-valued attributes . The same criterion -LRB- i.e. information gain -RRB- as used in binary splitting -LRB- Quinlan , 1986 -RRB- is employed ; however , multivalue splitting is realised by recursively applying the same criterion to the subsets of the first split and so on . A stopping criterion which is based on the minimum description length principle , decides when to stop splitting . This criterion -LRB- Fayyad & Irani , 1993 -RRB- is re-stated as follows : MDLPC Criterion : The partition induced by a cut point T of attribute A for a set S of N examples is accepted iff Gain -LRB- A ; T ; S -RRB- > log2 -LRB- N ? 1 -RRB- N + ? -LRB- A ; T ; S -RRB- N and it is rejected otherwise . Where ? -LRB- A ; T ; S -RRB- = log2 -LRB- 3k ? 2 -RRB- ? -LSB- kEnt -LRB- S -RRB- ? k1Ent -LRB- S1 -RRB- ? k2Ent -LRB- S2 -RRB- -RSB- Gain -LRB- A , T ; S -RRB- is the information gain of a cut point T. Ent -LRB- S -RRB- is the class entropy of a subset S and k is the number of classes in subset S. Another discretization method is motivated by unsupervised learning . Van de Merckt -LRB- 1993 -RRB- makes use of a clustering method to convert continuousvalued attributes into binary splits . Two methods are used ; the first is a direct application of clustering method and the second is based on the first criterion but modified with the entropy measure . The two criteria are re-stated as follows : i. Unsupervised Monothetic Constrast Criterion : Contrast -LRB- N1 ; N2 ; A -RRB- = N1N2 N1 + N2 -LRB- mA1 ? mA2 -RRB- 2 where N1 ; N2 are number of instances of the resulting binary split , mA1 is the mean value for attribute A of N1 instances . ii . Mixed Supervised/Unsupervised Monothetic Criterion : CE -LRB- N1 ; N2 ; A -RRB- = Contrast -LRB- N1 ; N2 ; A -RRB- Ent -LRB- S -RRB- when Ent -LRB- S -RRB- is zero , it is replaced by 10E-6 . The desirable split is the cut point that produces the highest Contrast or CE . We are now ready for the experiment . The experimental design is explained in the following section and the empirical results are shown in Section 5 . 4 Experimental Design The experiment is designed to verify the hypothesis : The performance of IB1-MVDM can be up-graded by using a proper method of converting continuous-valued attributes to nominal attributes at the outset , and this algorithm compares favourably with IB1 and C4 .5 in most domains . First , a proper conversion method is sought . Six methods have been tried , which include the three methods mentioned in Section 3 and a combination of these methods . Details of the conversion methods are listed below , where in each case , continuous-valued attributes are converted to : a. twenty equal interval nominal attributes . b. binary attributes according to Unsupervised Monothetic Contrast Criterion -LRB- Van de Merckt , 1993 -RRB- . c. binary attributes according to Mixed supervised/Unsupervised Monothetic Criterion -LRB- Van de Merckt , 1993 -RRB- . d. nominal attributes according to MDLPC Criterion , -LRB- Fayyad & Irani , 1993 -RRB- e. nominal attributes according to Mixed supervised/Unsupervised Monothetic Criterion with MDLPC Criterion as the stopping criterion . f. nominal attributes according to Unsupervised Monothetic Contrast Criterion with MDLPC Criterion as the stopping criterion . After experimenting with the above six methods , the best method is chosen in combination with IB1-MVDM ; and it is designated as IB1-MVDM * . A further experiment is designed to compare IB1-MVDM * with IB1-MVDM , IB1 and C4 .5 , where all algorithms will be tested without discretization of continuous-valued attributes except IB1-MVDM * . We are also interested in finding out what are the effects of the discretization on these algorithms . A comparison is made between running with and without discretization for each algorithm . Algorithms run with data conversion are postfixed with \* '' to differentiate them from those run without . 5 Empirical Results The characteristics of the experimental domains and the performance of C4 .5 are given in Table 2 . These domains include the twelve benchmark domains from Zheng -LRB- 1993 -RRB- . In addition , two continuous-valued domains -LRB- heart disease cleveland and glass -RRB- , one mixed attribute types domain -LRB- automobile -RRB- and a nominal attributes domain -LRB- nucleotides and codons ; see explanation below -RRB- are added to the list . Domain % Err C4 .5 # Ex # Classes # Attr & Type breast cancer -LRB- bcw -RRB- 5.4 % 699 2 9C diabetes 29.3 % 768 2 8C waveform 30.8 % 300 3 40C heart disease 25.9 % 303 2 13C glass 32.3 % 214 6 9C hypothyroid 0.8 % 3163 2 18B +7 C hepatitis 23.5 % 155 2 13B +6 C automobile 24.1 % 205 7 4B +6 N +15 C lymphography 22.1 % 148 4 9B +9 N promoter 22.6 % 106 2 57N soybean 8.1 % 683 19 16B +19 N monks-2 35.0 % 169-432 2 2B +4 N nettalk -LRB- phoneme -RRB- 19.7 % 5438 52 7N led7 28.6 % 200 10 7B led24 37.9 % 200 10 24B nucleotides -LRB- b -RRB- 37.0 % 20000 2 60B nucleotides -LRB- n -RRB- 35.3 % 20000 2 15N codons -LRB- b -RRB- 25.7 % 20000 2 320B codons -LRB- n -RRB- 36.0 % 20000 2 5N B : Binary , N : Nominal , C : Continuous . Table 2 : Details of experimental domains and performance of C4 .5 All these domains are obtained from the UCI Machine Learning Databases except the waveform , led7 and led24 domains , and the last four datasets . These four datasets are actually produced from a single promoter domain , first used by Craven & Shavlik -LRB- 1993 -RRB- . The original representation is nucleotides -LRB- n -RRB- which has fifteen 4-value attributes ; nucleotides -LRB- b -RRB- is the binary conversion of the original data . Codons -LRB- n -RRB- 1 is a transformation from the 1Three consecutive nucleotides form a codon -LRB- Craven & Shavlik , 1993 -RRB- . original data , which has five 64-value attributes and codons -LRB- b -RRB- is the binary version . The first set of eight domains is those with continuous-valued attributes ; the first five are all continuous-valued attributes domains and the other three have both types of attributes . The second set constitutes seven all nominal attributes datasets and four binary attributes datasets . The first set of eight domains will be used to verify the hypothesis and finally a complete results on all nineteen datasets will be provided . All experiments , except in the monks-2 , nucleotides and codons datasets , are conducted over 50 runs with randomly selecting 90 % of the instances for training and using the remaining 10 % as test data . Because a separate set of training data -LRB- 169 -RRB- and testing data -LRB- 432 -RRB- are given in the monks-2 domain , it is conducted only one run . In the nucleotides and codons datasets , each dataset is split into four equal size and non-overlapping subsets . Four runs are conducted in each of these datasets using each subset -LRB- 5000 -RRB- as training data and one of the other subset -LRB- 5000 -RRB- as testing data . Only the average results for these runs will be given . For experiments that employ discretization of continuous-valued attributes in the front-end , they are conducted using the training data first , and then the test data is converted according to the cut points obtained from the training data conversion . All results shown in the following section are with reference to the error rates of C4 .5 , tabulated in Table 2 , unless otherwise stated . 5.1 Selecting A Discretization Method Table 3 shows the results of IB1-MVDM * with each of the six discretization methods described in Section 4 . The best method in each domain is highlighted . The negative figure indicates that IB1-MVDM * performs better than C4 .5 by that percentage and vice versa . Method -LRB- d -RRB- achieves four best results and close to the best in the other four domains . Method -LRB- f -RRB- achieves three best results but performs poorly in the diabetes and glass domains . Method -LRB- c -RRB- has the best result in the diabetes domain but performs relatively poor in three domains . The other three methods do not produce any best result but have poor performance in at least three domains . Thus , method -LRB- d -RRB- is chosen to be the method used in combination with IB1-MVDM in the following experiments . These results also provide a message that there is a discretization method which will make IB1-MVDM * performs better than IB1-MVDM . Discretization Method Domain IB1-MVDM a b c d e f bcw -0.9 -1.2 -1.4 -0.7 -0.8 -1.1 -1.5 diabetes +0.5 +6.5 -0.2 -1.4 -0.7 +1.9 +0.9 waveform +5.7 -3.2 -8.2 -6.3 -8.2 -8.1 -10.1 heart -1.2 -4.2 +3.9 -3.8 -6.1 -4.6 -3.0 glass -2.3 +25.1 +14.4 +12.9 -3.5 +7.1 +9.3 hypothyroid +71.6 +4.5 +2.9 +1.4 +0.8 +1.9 +1.2 hepatitis -4.4 -2.6 -0.6 -3.5 -3.2 -3.2 -5.4 automobile +0.6 +26.6 +6.0 +4.4 -8.5 -6.3 -7.2 Table 3 : Results of six types of conversion and IB1-MVDM 5.2 Performance Comparison The performance results for IB1-MVDM * , IB1-MVDM and IB1 -LRB- with reference to C4 .5 -RRB- are shown in Table 4 . The second column indicates that IB1-MVDM * outperforms C4 .5 in seven out of eight domains and performs marginally poorer in the hypothyroid domain . Substantial improvements are observed in the waveform , heart and automobile domains . With comparison to IB1-MVDM , IB1-MVDM * is a clear winner in six domains and obtains marginally poorer results in the bcw and hepatitis domains . The same trend is observed in IB1 since IB1 and IB1-MVDM are virtually the same algorithm in the all continuous-valued domains . This experiment supports our hypothesis that IB1-MVDM * is the best algorithm among these four algorithms in most of the domains tested . The extremely poor performance of IB1-MVDM in the hypothyroid domain needs special attention . An analysis on the MVDM produced for the -LRB- original -RRB- nine binary attributes indicates that all the value-differences are less than 0.01 . When these value-differences are used in conjunction with the continuous-valued attributes in the distance calculation , they have virtually no impact . Thus , the nine binary attributes are deemed irrelevant in IB1 - MVDM . Because of the discretization in IB1-MVDM * , the value-differences for the -LRB- original -RRB- continuous-valued attributes have comparable \ weight '' to that of the binary attributes . As a result , all eighteen attributes play a part in classification . In fact , in this domain , any of the discretization methods would do better than IB1-MVDM . Domain IB1-MVDM * IB1-MVDM IB1 IB1 * C4 .5 * bcw -0.8 -0.9 -0.9 -2.1 -0.2 diabetes -0.7 +0.5 +0.5 -0.5 -3.5 waveform -8.2 +5.7 +5.7 -0.2 +5.5 heart -6.1 -1.2 -1.2 -4.9 -2.8 glass -3.5 -2.3 -2.3 -5.5 -3.3 hypothyroid +0.8 +71.6 +1.9 +0.6 +0.3 hepatitis -3.2 -4.4 -4.2 -2.9 -1.2 automobile -8.5 +0.6 +0.1 -5.3 +0.9 Table 4 : Performance comparison In order to have a clearer picture in comparing the performance difference between the use of converted and original data , Table 5 is produced from the results in Table 4 by subtracting the results of using the converted data from the results of using the original data . Negative figures in Table 5 indicate that the algorithm which uses the converted data is better than that using the original data , and vice versa . The results clearly show that using the converted data is more desirable for all three algorithms . IB1-MVDM * achieves the greatest improvement , and there is a modest one for IB1 * . A mixed performance is observed in C4 .5 * slightly favouring the converted data with more domains achieving improved results . As a result of this comparison , we will only present results of all algorithms that are combined with discretization method -LRB- d -RRB- in the next section . Domain IB1-MVDM IB1 C4 .5 bcw +0.1 -1.2 -0.2 diabetes -1.2 -1.0 -3.5 waveform -13.9 -5.9 +5.5 heart -4.9 -3.7 -2.8 glass -1.3 -3.2 -3.3 hypothyroid -70.8 -1.3 +0.3 hepatitis +1.2 +1.3 -1.2 automobile -9.1 -5.4 +0.9 Table 5 : Performance difference between converted and original data for three algorithms 5.3 Overall Results Table 6 shows the results of all domains for IB1-MVDM * , IB1 * , NL * and C4 .5 * . NL is the naive Bayesian learner that uses the m-estimate -LRB- Cestnik , 1990 -RRB- . NL * is a simple classifier but often has good performance ; its results are given here for the purpose of comparison . Again , the results shown are with reference to the error rates of C4 .5 and the best results are highlighted . Domain IB1-MVDM * NL * IB1 * C4 .5 * bcw -0.8 -2.5 -2.1 -0.2 diabetes -0.7 -3.8 -0.5 -3.5 waveform -8.2 -9.6 -0.2 +5.5 heart -6.1 -7.7 -4.9 -2.8 glass -3.5 -1.7 -5.5 -3.3 hypothyroid +0.8 +0.6 +0.6 +0.3 hepatitis -3.2 -8.1 -2.9 -1.2 automobile -8.5 +6.6 -5.3 +0.9 lymphography -5.7 -3.4 +0.4 0.0 promoter -15.5 -13.7 -3.3 0.0 soybean -2.1 +6.2 +0.4 0.0 monks-2 -9.5 +3.2 -5.4 0.0 nettalk -LRB- phoneme -RRB- -3.5 +5.3 +6.4 0.0 led7 +4.0 +0.3 +1.6 0.0 led24 +0.4 +0.1 +31.5 0.0 nucleotides -LRB- b -RRB- -0.3 -8.1 -3.0 0.0 nucleotides -LRB- n -RRB- +2.4 -6.2 -1.3 0.0 codons -LRB- b -RRB- +5.5 -4.0 +4.5 0.0 codons -LRB- n -RRB- -6.4 -14.4 -5.8 0.0 Table 6 : Overall results C4 .5 * has the best results in the hypothyroid , led7 and led24 domains ; but the gain is marginal . IB1-MVDM * achieves the best results in six domains -LRB- automobile , lymphography , promoter , soybean , monks-2 and nettalk -RRB- . IB1 * obtains the best result only in the glass domain . NL * attains the best results in six domains or nine datasets -LRB- the last four datasets are only different representations of a single domain -RRB- . NL * 's performance seems to swing at two extremes ; in some domains it achieves exceptional good accuracies , e.g. in the hepatitis , nucleotides and codons datasets . Whereas in the automobile , monks-2 and soybean domains , it obtains the worst results . IB1 * has moderate performance , i.e. in eleven domains -LRB- or fourteen datasets -RRB- , it has performance in between the best and the worst . IB1-MVDM * outperforms C4 .5 in fourteen out of the nineteen datasets . Considerable big improvements are attained in seven datasets , i.e. the waveform , heart , automobile , lymphography , promoter , monks-2 and codons -LRB- n -RRB- datasets . The results also show that IB1-MVDM * does n't always perform better than IB1 * in nominal attributes domains . Examples are the bcw , glass -LRB- after discretization -RRB- and nucleotides datasets ; and IB1-MVDM * seems to have problems in binary domains as well . Looking from the other perspective , IB1-MVDM * seems to deal with irrelevant attributes quite well . This can be seen in the two artificial domains with irrelevant attributes -LRB- i.e. led24 and waveform -RRB- , where substantial gains are achieved when compared with IB1 * . Overall , IB1-MVDM * is clearly the best choice among these four algorithms . 6 Discussion and Future Work The empirical results show that a simple approach in instance-based learning that uses two separate metrics for continuous-valued and nominal attributes can bring about poor performance -LRB- an example is in the hypothyroid domain -RRB- . Converting continuous-valued attributes to nominal attributes at the outset is demonstrated to be a simple yet effective approach . In this way , IBL uses only one metric rather than two separate metrics for continuous-valued and nominal attributes . It provides a coherent approach to the treatment of these two types of attributes . MVDM seems to be a good metric to deal with irrelevant attributes . Because MVDM establishes that two values -LRB- of the same attribute -RRB- are similar if they occur with the same relative frequency for all classifications , it provides a means to tolerate irrelevant attributes . Aha -LRB- 1990 -RRB- uses an extra parameter , namely weight for each attribute to express the attribute 's relevancy ; however , MVDM turns out to be a more coherent method which incorporates the capability in the distance metric . The value-difference between two values approaches zero if the attribute is random as they have about the same relative frequency for all classifications . Cost and Salzberg 's claim that the valued-difference metric is superior to the \ overlap '' metric , is generally true ; but there are some cases where the opposite is true . It would be interesting to find out what is the reason and thus possibly provide clues to refine the value-difference metric . Fayyad and Irani 's conversion method emerges as the best in the experiments . However , a clustering method also shows promise . Nominal conversion -LRB- f -RRB- achieves the best results in three domains -LRB- bcw , hepatitis and waveform -RRB- . This suggests further investigation of unsupervised methods should be carried out . Though we directly employ Fayyad and Irani 's formulation , it is used in a rather different way ; we use the formulation only once for the full set of training data , whereas Fayyad and Irani apply it at each level and each branch of the trees . The experiment that compares the performance difference between converted and original data , indicates that converted data is more desirable for IB1-MVDM , IB1 and C4 .5 . As Fayyad & Irani -LRB- 1993 -RRB- and Van de Merckt -LRB- 1993 -RRB- conduct discretization at each level and each branch of the decision trees , it would be interesting to see how these results compare with the discretization done at the outset in the framework of decision trees . Catlett -LRB- 1991 -RRB- performed such a comparison by using a discretization method similar to that used by Fayyad & Irani -LRB- 1993 -RRB- , and found that the combined discretization method and C4 -LRB- an earlier version of C4 .5 -RRB- achieve little or no loss of accuracy -LRB- and sometimes increased accuracy -RRB- over C4 but dramatically increase in efficiency . This findings coincide with the results of our experiment using C4 .5 . 7 Conclusion This paper explores a method that combines two recent techniques in machine learning and produces a learning algorithm that is superior to IB1 , IB1-MVDM and C4 .5 in most of the domains tested . The experiments reported here also identify two areas where IB1-MVDM * could be improved . One area is the refinement of the value-difference metric and the other is the discretization method . Acknowledgements This research was partially supported by an Australia Research Council grant -LRB- to J.R.Quinlan -RRB- and by a research agreement with Digital Equipment Corporation . This author is partially supported by Equity and Merit Schol - arship Scheme . Numerous discussions with J.R. Quinlan , Mike Cameron Jones , Zijian Zheng , and Alen Varsek have been very helpful . Thanks to J.R. Quinlan and David W. Aha for providing C4 .5 and IB1 algorithms . We are also grateful to P.M. Murphy and D.W. Aha for creating and managing the UCI Repository of Machine Learning Databases & Domain Theories Appendix A We have attempted to modify the MVDM by using the m-estimate , as used in the naive Bayesian learner -LRB- Cestnik , 1990 -RRB- . -LRB- the frequency ratios are modified using the m-estimate by setting m = 2 -RRB- . We repeated the experiment and the relative results of using and not using the m-estimate modification are tabulated in Table A1 . A negative figure indicates MVDM with the m-estimate modification is better and vice versa . The results show minor variations with more domains attaining marginal gain than loss . Domain IB1-MVDM IB1-MVDM * bcw 0.0 0.0 diabetes 0.0 +0.2 waveform 0.0 -0.9 heart 0.0 -0.1 glass 0.0 -1.0 hypothyroid 0.0 0.0 hepatitis -0.3 0.0 automobile -1.7 -0.4 lymphography +1.3 promoter -0.4 soybean 0.0 monks-2 0.0 nettalk -LRB- phoneme -RRB- -0.3 led7 0.0 led24 +0.1 nucleotides -LRB- b -RRB- 0.0 nucleotides -LRB- n -RRB- +0.1 codons -LRB- b -RRB- -0.1 codons -LRB- n -RRB- 0.0 Table A1 : Results of relative performance of m-estimate modification in MVDM Appendix B Properties of MVDM described by Cost & Salzberg -LRB- 1992 -RRB- are : a. Two values -LRB- of an attribute -RRB- are similar if they occur with the same frequency for all classifications . b. Mathematical properties : 1 : ffi -LRB- a ; b -RRB- > = 0 ; for a < > b : 2 : ffi -LRB- a ; b -RRB- = ffi -LRB- b ; a -RRB- : 3 : ffi -LRB- a ; a -RRB- = : 4 : ffi -LRB- a ; b -RRB- + ffi -LRB- b ; c -RRB- > = ffi -LRB- a ; c -RRB- : There is one important property of MVDM which is not mentioned by Cost & Salzberg -LRB- 1992 -RRB- , i.e. MVDM is a means to measure attribute relevancy . The following two conditions describe the situations where an attribute becomes irrelevant . a. f ffip -LRB- vi ; vj -RRB- ' j i < > j g b. there exists some k and l with k < > l such that ffiq -LRB- vk ; vl -RRB- AE max -LRB- f ffip -LRB- vi ; vj -RRB- > j i < > j g -RRB- The first condition states that if all value-differences of an attribute , p , approximate or equal to zero , then this attribute is an irrelevant attribute . The second condition states that if there exists some value-differences of attribute q that are a lot greater than the maximum value-difference of attribute p , then attribute p would become irrelevant . References Aha , D.W. -LRB- 1990 -RRB- , A Study of Instance-Based Algorithms for Supervised Learning Tasks , PhD Thesis , Department of Information and Computer Science , University of California , Irvine , Technical Report 90-42 . Aha , D.W. , D. Kibler & M.K. Albert -LRB- 1991 -RRB- , Instance-Based Learning Algorithms , Machine Learning , 6 , pp.37-66 . Catlett , J. -LRB- 1991 -RRB- , On Changing Continuous Attributes into Ordered Discrete Attributes , In European Working Session on Learning . Cestnik , B. -LRB- 1990 -RRB- , Estimating Probabilities : A Crucial Task in Machine Learning , in Proceedings of the European Conference on Artificial Intelligence , pp. 147-149 . Cost , S & S. Salzberg -LRB- 1993 -RRB- , A Weighted Nearest Neighbor Algorithm for Learning with Symbolic Features , Machine Learning , 10 , pp.57-78 . Craven & Shavlik -LRB- 1993 -RRB- , Learning to Represent Codons : A Challenge Problem for Constructive Induction , Proceedings of 13th International Joint Conference on Artificial Intelligence , pp.1319-1324 , Morgan Kaufmann . Fayyad , U.M. & Irani , K.B. -LRB- 1993 -RRB- , Multi-Interval Discretization of ContinuousValued Attributes for Classification Learning , in Proceedings of 13th International Joint Conference on Artificial Intelligence , pp.1022-1027 , Morgan Kaufmann . Quinlan , J.R. -LRB- 1986 -RRB- , Induction of Decision Trees , Machine Learning , 1 , 81-106 . Quinlan , J.R. -LRB- 1993 -RRB- , C4 .5 : Program for machine learning , Morgan Kaufmann . Stanfill , C. & Waltz , D. -LRB- 1986 -RRB- , Toward Memory-Based Reasoning , in Communications of the ACM , vol .29 , No. 12 , pp. 1213-1228 . Van de Merckt , T. -LRB- 1993 -RRB- , Decision Trees in Numerical Attributes Spaces , in Proceedings of 13th International Joint Conference on Artificial Intelligence , pp. 1016-1021 , Morgan Kaufmann . Zheng , Z. -LRB- 1993 -RRB- , A Benchmark for Classifier Learning , in Proceedings of the Sixth Australian Joint Conference on Artificial Intelligence , pp. 281-286 , World Scientific .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

bayesian learning
instance-based learning