Abducing memory barriers

A concurrent work-stealing algorithm is analysed to discover necessary positions for memory barriers. The analysis is targeted at the Power processor, and takes into account both instruction-execution reordering and delayed writes. Formal verifications of the effectiveness of the barriers under instructionexecution reordering are developed, and a semi-formal argument (using formal assertions with semantic reasoning) verifies barriers which deal with inter-thread propagation effects. The published algorithm had a number of suggested barriers to guard against reordering; it is shown that it needs one more barrier for that purpose, and two more to deal with propagation effects. The effect of missing barriers is described. Programming is in a weird place. Our programs are mostly sequential but hardware is mostly parallel, even within a single processor. Caches, pipelines, weak memory and speculative execution conspire to rearrange execution in surprising ways. A single isolated program keeps its sequential meaning, but shared-variable concurrency becomes rather tricky. Special barrier instructions are needed to enforce some discipline, and it’s hard to understand when and where they are needed. There’s been a lot of study of the hardware [3, 26, 25] and some use of model-checking to verify barrier placement [13, 17, 5]; this paper looks at the problem from a programmer’s point of view and uses verification. void put(Task tv) { Order write at 4 before write at 5 1: h := Hd; 2: t := Tl ; 3: if t = h+Ta.size then { expand(); goto 1 }; 4: Ta.array[ t%Ta.size] := tv ; 5: Tl := t+1 } Task take() { local Task tv ; 1: h := Hd; 2: t := Tl ; 3: if h=t then return EMPTY; 4: tv := Ta.array[h%Ta.size]; 5: Hd := h+1; 6: return tv } void expand() { Order writes in 2 and 4 before write in 5 Order write in 5 before write in put:5 1: s := Ta.size; 2: a’ := new SizeTaskArray(2∗s); 3: for i = Hd:Tl-1, 4: a ’. array[ i%a’. size] := Ta.array[ i%Ta.size]; 5: Ta := a’ } Task steal () { Order read in 1 before read in 2 Order read in 1 before read in 4 Order read in 5 before CAS in 6 local Task tv ; 1: h := Hd; 2: t := Tl ; 3: if h=t then return EMPTY; 4: a := Ta; 5: tv := a.array[h%a.size]; 6: if CAS(&Hd,h,h+1) then goto 1; 7: return tv ; } Figure 1: Michael et al.’s FIFO IWSQ (from [20] with abbreviated names) Concurrent algorithms are often published as if they were to be executed on a sequentially-consistent machine and the placement of barriers is treated as if it were merely an implementation detail. This won’t do: barriers are an essential part of an algorithm designed to run fast and concurrently. In figure 1, taken from [20], the authors have indicated where the barriers ought to go (though in private correspondence [30] they made it clear that these are only suggestions, not tested on any particular current processor; their experiments were on an architecture which doesn’t reorder execution). In this paper we discover the barriers that are needed by logical argument. We take each procedure and draw a dependency graph of its commands, showing which must follow what, even if the machine reorders execution. Then, by trying to make a formal proof that the graph preserves some global invariants, we find places where the proof won’t go through. We fix those problems by inserting an extra ordering, provided by a barrier instruction. This is a sort of abduction [23] in that it involves guessing and verifying. We find all the suggested barriers and one extra. Then by considering orderings between different procedures we find two more barriers. Missing barriers cause bugs, and we describe the bugs that result. Our technique works well for our chosen example. That is satisfying, but it is work in progress: our dependency graphs aren’t formally tied to the semantics of the machine; when considering memory propagation effects we reason semi-formally; and it would be easy to guess too many barriers. For all that, we do find barriers, and it ought to be possible in the near future to dot all the logical i’s and cross the formal t’s. Related work is either on automatically placing barriers [19, 2, 1] or removing redundant ones [29]. Most of it assumes TSO, the execution model of the Intel 86; for weaker architectures, such as IBM Power, the placement problem seems to be undecidable [7, 6]. It remains to be seen if automatic techniques can discover all and only the barriers needed for FIFO IWSQ on a weak architecture. 1 An example algorithm Work stealing [4, 14] is a technique for scheduling tasks on multiprocessor machines. Each scheduler thread manages its own queue of tasks, but when the queue is empty it may try to steal a task from another. Michael et al. [20] describe an ‘idempotent work stealing’ algorithm (henceforth IWSQ): we analyse their