The Performance Impact of Granularity Control and Functional Parallelism ? Jos?e E. Moreiray Dale Schoutenz Constantine Polychronopoulosz moreira@watson.ibm.com fschouten,cdpg@csrd.uiuc.edu y IBM T. J. Watson Research Center Yorktown Heights , NY 10598-0218 z Center for Supercomputing Research and Development , Coordinated Science Laboratory University of Illinois at Urbana-Champaign 1308 W. Main St. Urbana , IL 61801-2307 CSRD Technical Report 1449 presented at the Eighth Workshop on Languages and Compilers for Parallel Computing Proceedings to be published by Springer-Verlag Abstract . Task granularity and functional parallelism are fundamental issues in the optimization of parallel programs . Appropriate granularity for exploitation of parallelism is affected by characteristics of both the program and the execution environment . In this paper we demonstrate the efficacy of dynamic granularity control . The scheme we propose uses dynamic runtime information to select the task size of exploited parallelism at various stages of the execution of a program . We also demonstrate that functional parallelism can be an important factor in improving the performance of parallel programs , both in the presence and absence of loop-level parallelism . Functional parallelism can increase the amount of large-grain parallelism as well as provide finer-grain parallelism that leads to better load balance . Analytical models and benchmark results quantify the impact of granularity control and functional parallelism . The underlying implementation for this research is a low-overhead threads model based on user-level scheduling . 1 Introduction The magnitude to which runtime overhead affects performance has been widely demonstrated -LSB- 2 , 3 , 12 -RSB- . In order to alleviate this problem -LSB- 12 -RSB- and other subsequent studies provided an environment that allows the user to control the number of parallel tasks a given parallel application generates . Given a fixed number of resources , a user or ? This work was supported by the Office of Naval Research under grant N00014-94-1-0234 . Computational facilities were provided by the National Center for Supercomputing Applications . Jos?e Moreira was at the University of Illinois during the development of this research . compiler can restrict the maximum number of parallel tasks of a parallel application to less than or equal to a predetermined amount . This paper reports on an implementation which employs the notion of dynamic granularity control . At any given time , the number of parallel activities a process generates is proportional to the number of physical resources allocated to that process . This allows the operating system to dynamically allocate a varying number of processors to different processes . In fact , the number of processors allocated to a particular process may vary over its lifetime . The immediate impact of granularity control is the elimination of unnecessary overhead due to frequent context switching , creation and scheduling of tasks , additional interprocessor communication , and increased memory latency . Our method relies on a program representation which encapsulates the hierarchy of computations inherent in a parallel application . This allows for parallelism to be exploited first at the highest level of this hierarchy which corresponds to the outermost loops and the first-level function calls . Subject to resource availability , inner levels of parallelism are exploited by decomposing nested parallelism . A related focus of this work is the performance implications of the exploitation of functional -LRB- nonloop -RRB- parallelism . Our experiments indicate that functional parallelism can improve performance by a significant margin , even in situations where data -LRB- loop -RRB- parallelism is in abundance . This paper is organized as follows : Section 2 describes the programming model and target machine architecture . Section 3 describes an autoscheduling threads model , nanoThreads . Queue management and granularity control issues are addressed in Sections 4 and 5 respectively . The environment used for our measurements is described in Section 6 . An analytical model showing the benefits of the exploitation of functional parallelism and experimental results from synthetic benchmarks are presented in Section 7 . The set of benchmarks used for more general measurements is listed in Section 8 , and the results from these measurements are shown in Section 9 . Finally , related work is discussed in Section 10 and concluding remarks are given in Section 11 . 2 Machine and Programming Model The target machine model is a shared address space multiprocessor with a multiprogramming environment . Therefore only a subset of the machine 's processors will be allocated to a particular program . We call this subset of processors a partition and let this partition be time-variant , meaning that processors may be added or removed by the operating system during the execution of the job . The program model is the hierarchical task graph -LSB- 10 -RSB- , or HTG , combined with autoscheduling -LSB- 19 -RSB- . The HTG is an intermediate program representation that encapsulates data and control dependences at various levels of task granularity . This structure is used to generate autoscheduling code , which includes the scheduling operations directly within the program code . The HTG represents a program in a hierarchical structure , thus facilitating task-granularity control . Information on control and data dependences allows the exploitation of functional -LRB- task-level -RRB- parallelism , in addition to data -LRB- loop-level -RRB- parallelism . A brief summary of the properties of the HTG is given here and details can be found in -LSB- 3 , 8 , 9 , 10 , 19 -RSB- . The hierarchical task graph is a directed acyclic graph HTG = -LRB- HV ; HE -RRB- with unique nodes START and STOP 2 HV , the set of vertices . Its edges , HE , are a union of control -LRB- HC -RRB- and data dependence -LRB- HD -RRB- arcs : HE = HC -LSB- HD . The nodes represent program tasks and can be of three types : simple , compound , and loop . A simple node represents the smallest schedulable unit of computation . A compound node is an acyclic task graph -LRB- ATG -RRB- comprised of smaller nodes , each recursively defined as an HTG . A loop node represents a task that is either a serial loop -LRB- in which case all iterations must be executed in order -RRB- or a parallel loop -LRB- in which case the iterations may be executed simultaneously in any order -RRB- . The body of a loop can be an HTG . An HTG may have local variables that can be accessed by any task in the HTG . In the general model , each task may have internal task-local variables . Each HTG is associated with a set of boolean flags -LRB- local variables -RRB- that mark the execution of nodes and arcs in the HTG . As a task executes , its autoscheduling drive code at the exit block of the task updates the values of these boolean flags . Each node is associated with an execution tag : an expression on the boolean flags that is derived from the data and control dependences and represents the execution condition for that node . Let '' -LRB- x -RRB- denote the execution tag of node x. Whenever the values of the boolean flags cause '' -LRB- x -RRB- to evaluate to TRUE , node x will be enabled and ready to execute . This evaluation of '' -LRB- x -RRB- is also performed by the drive code for all the successors of a task and the enabled tasks are placed in a task queue . Autoscheduling and the HTG effectively implement a macro dataflow model of execution on a conventional multiprocessor . 3 NanoThreads NanoThreads -LSB- 18 -RSB- is a threads architecture that combines low-overhead threads and autoscheduling . Each nanoThread corresponds to an HTG task . Since the tasks effectively schedule themselves , via the exit blocks of the HTG , the user-level scheduler is a simple loop . Given a pointer to a task queue , it retrieves tasks from the queue and executes them directly rather than performing a context switch . Each processor in the partition allocated to a job runs such a scheduling loop . Only one system-level thread or shared memory process per processor is needed , and it only has to be created at the beginning of the job execution . As soon as the system-level thread starts running , it enters the scheduling loop and then all the user-level scheduling is done by the drive code of autoscheduling . We call these system-level threads virtual processors since they are doing the actual work . Every virtual processor attached to a particular job accesses the same task queue . Each task in the queue contains two pieces of information : a code pointer -LRB- program counter -RRB- and a pointer to an activation frame -LRB- AF -RRB- containing the data local to that task . The activation frames are organized in a tree-structure called the cactus-stack . This is illustrated in Figure 1 , together with the main scheduling loop . An entire parallel loop is represented by a single task descriptor which , in addition to the above information , contains a loop iteration counter . This allows the processors to perform dynamic loop scheduling . The overhead associated with task dispatch is small , consisting of loading a register with the activation frame pointer and jumping to the beginning of the code for the task . P0P1P2P3P CAF ... RootF rame 1F rame 2F rame 3F rame n Cactus Stack Task QueueP rocessors Scheduling loop repeat fetch task T execute T until DONE Fig. 1 . Task queue , cactus stack , and main scheduling loop for a nanoThreaded program . Other overhead of enqueueing tasks and allocating activation frames is incurred in the program code , specifically in the entry and exit blocks of the nodes of the HTG . 4 Queue Management Processors dispatch ready tasks and enqueue new tasks from and onto a task queue respectively . A centralized queue provides good load balancing , but it can also be a source of contention . In addition , a centralized queue does not facilitate preferential assignment of tasks to processors . A distributed task queue is used to address both the contention and the locality issues . Each processor has its own local queue , and the enqueueing and dequeueing policies work as follows : ? Enqueueing : When a processor determines that a new task has become ready for execution , the corresponding task descriptor is placed in the processor 's local queue . ? Dequeueing : A processor first tries to fetch a task from its own local queue . If its queue is empty , it then searches the local queues of other processors for work . On a heavily loaded system , local tasks tend to cluster on the same processor , enhancing locality of access . Consider the loop nest : doall I = 1 ; N doall J = 1 ; M Body end end Assume the descriptor for the doall I is in the local queue of processor P0 . Other idle processors will fetch from P0 's queue and participate in the execution of doall I . The execution of one iteration of doall I by processor Pi will create the corresponding descriptor for one instance of doall J in the local queue of Pi . While there are enough iterations of doall I to keep all processors busy , each processor will be fetching from its own local queue . Thus , contention is kept to a minimum and locality inside each iteration of doall I can be preserved . When the iterations of doall I are exhausted -LRB- i.e. they have all been issued -RRB- idle processors will start participating in the execution of remote instances of doall J , thus trading locality for load balance . 5 Granularity Control Determining the best task granularity is one of the fundamental optimization problems in parallel processing . The granularity or grain size of a task is informally used to indicate the size of the task . Typically the overhead of creating and scheduling a task is approximately the same regardless of its execution time . A large grain task spends a smaller percentage of time performing system functions such as scheduling and allocating its activation frame . The efficiency may be higher with larger tasks , thus reducing the relative overhead of parallelism . On the other hand , maximum exploitation of parallelism naturally leads to fine granularity tasks which can facilitate load balancing and increased utilization at the expense of efficiency . The smaller the overhead , the finer the granularity that can be exploited effectively . However , even in systems that support fine granularity , such as nanoThreads , adjusting task granularity dynamically may be beneficial , and in fact necessary , under certain conditions , as supported by our experimental results . Granularity control in nanoThreads is based on hybrid scheme , part static and part dynamic , as explained below . The main goal of the static part is to guarantee a minimum task size at all times , while the dynamic part uses runtime information to select the appropriate level of granularity to exploit -LRB- i.e. above the minimum set by the compiler , and up to the maximum present in the application -RRB- . 5.1 Static Granularity Control Compile-time analysis can be used to guarantee that no task in the HTG is smaller than a certain size . The minimum size depends on the per task overhead of the system . The details of selecting an appropriate minimum size are beyond the scope of this paper . Static minimum granularity control can be implemented through task merging , a process described in -LSB- 3 -RSB- . Another aspect of static granularity control is to help prevent unnecessarily conservative dynamic granularity control decisions . When the overhead for task scheduling is negligible , as compared to the task size , there is little advantage in serializing the task . On the other hand , a dynamic granularity control decision that inhibits the parallelism of such a large task can only have a negative effect . Using static granularity control to prevent serialization can enhance performance . Our current implementation does not include automatic static granularity control , but does include facilities for specifying it manually . 5.2 Dynamic Granularity Control Compile-time preconditioning of a program enforces a minimum granularity . In our scheme , however , the exact task size is determined at runtime and depends on program properties as well as runtime conditions . The latter depends largely on workload characteristics . Modern processor scheduling systems , such as Process Control -LSB- 12 -RSB- and Scheduler Activations -LSB- 2 -RSB- , space-share the processors in a multiprocessor to improve utilization and locality of reference . In this case , the number of processors allocated to a particular job depends on other jobs running at the same time . Dynamic granularity control works by exploiting parallelism beginning with very coarse tasks -LRB- outermost loops or first-level calls -RRB- and progressively moving to the inner subtasks . The granularity control decision is made when a new level of hierarchy in the HTG is about to be exploited , that is , at the entry block of a parallel task -LRB- ATG or parallel loop -RRB- . The decision will result in setting the execution mode for a particular task : serial or parallel . If the serial mode is chosen , then the tasks comprising this parallel task will be executed sequentially by the processor assigned to the task . If the parallel mode is chosen , then the subtasks will be made available for parallel execution . The decision is local to each parallel task and does not restrict the decisions made at a lower level of the hierarchy . An ideal granularity control scheme decomposes parallel tasks into subtasks only to the extent that all processors are utilized . Generating more tasks creates unnecessary overhead . Generating fewer task results in underutilization . The heuristics for granularity control presented and evaluated in this paper attempt to approach this ideal behavior with very simple operations so that granularity control does not become a significant overhead by itself . In our current implementation , when execution enters a parallel task its execution mode is set by the following : mode = ae parallel if Qn =P < = serial otherwise -LRB- 1 -RRB- where Qn is the number of tasks in the queue , P is the number of processors currently assigned to the process , and is a given threshold . This mechanism responds to the runtime change in the number of processors appropriately . As the number of processors P increases , the ratio QnP decreases and more tasks are created , making more work available for the processors . Conversely , as the number of processors decreases , fewer tasks are made available . In general , the best value for is machine and application dependent , but experience indicates that values like = 1 or = 2 work well for a variety of cases . 5.3 Distributed Task Queues If the task queue is physically distributed , the total number of tasks on all queues is a valid measure of the system load . However , maintenance of this information requires updates of a centralized counter with each task enqueueing or dequeueing operation . This creates the same type of contention that the distributed queue seeks to avoid . A fully distributed scheme involves each processor checking only its local queue . The same criteria as in the centralized queue is used -LRB- Expression 1 -RRB- , but Qn now represents the size of a processor 's local queue and P = 1 . Since idle processors fetch tasks from the queue of other processors , the local queue of a processor contains some information about the state of the entire system . However , information is less accurate than in the case of a centralized queue . A fully distributed queue was employed in the nanoThreads library . 6 Measurements Environment NanoThreads was implemented on an SGI Challenge shared-memory multiprocessor -LSB- 7 -RSB- . The machine on which measurements were obtained was configured with 32 R4400 processors and 2 Gigabytes of main memory . Each processor had a local cache with hardware support for cache coherence . Two different versions of nanoThreads were implemented on the SGI Challenge . One is a C++ nanoThreads library that allows the user to describe the HTG structure of a program as a collection of C++ classes . The actual body of the tasks can be coded either in C++ or in Fortran . The other implementation is a nanoThreads compiler based on the Parafrase-2 compiler -LSB- 17 -RSB- . This compiler uses the HTG representation of a Fortran program to automatically generate C++ code with an embedded scheduler . In both cases , systemlevel shared-address processes are created at the beginning of the program execution to implement the virtual processors , while all the nanoThreads operations are performed by user-level code . The benchmark suite used to evaluate the performance impact of granularity control and functional parallelism consists of 1 -RRB- synthetic benchmarks designed to produce specific behaviors of parallelism , and 2 -RRB- application kernels that are representative of the behavior of actual programs . The SGI Challenge timer facilities were used to measure wall-clock execution times . The measurements were made in single user time whenever possible , or when the machine was lightly loaded . 7 Benefits of Functional Parallelism In this section we present performance models that quantify the improvements delivered by the exploitation of functional parallelism , even in the presence of abundant data or loop-level parallelism . We use the task graphs in Figure 2 to develop our models . We first consider the task graph shown in Figure 2 -LRB- a -RRB- , which illustrates functional parallelism with internal loop parallelism . All n tasks can execute concurrently . Let each task be identically composed of a sequential part of size q and a parallel part of size p. Assume that the overhead for parallel processing is negligible . The serial execution time of task Ti -LRB- tTi -RRB- and of the entire task graph -LRB- tS -RRB- are given by : tTi = q + p ; tS = n -LRB- q + p -RRB- : -LRB- 2 -RRB- If only loop parallelism is exploited the tasks are executed in sequence and the total parallel execution time is : tP -LRB- loop -RRB- = n -LRB- q + p P -RRB- = qn + pn P : -LRB- 3 -RRB- T1T2TnS TARTSTOP do i = 1 , q ? ... enddod oall i = 1 , p ? ... enddo -LRB- a -RRB- doall i = 1 ; P do j = 1 ; i n T T T START STOP ... 1 2 enddo enddoall -LRB- b -RRB- Fig. 2 . Task graphs used in the performance models . -LRB- a -RRB- Each task has a serial part and a parallel part and the tasks are independent . -LRB- b -RRB- Parallel loop with internal functional parallelism . Alternatively , the set of P processors could be distributed uniformly among the n tasks executing concurrently . Assuming that n divides P , the parallel execution time on P processors is the same as the execution time of a task on P = n processors : tP -LRB- fp -RRB- = q + p P = n = q + pn P : -LRB- 4 -RRB- We define the functional speedup SP -LRB- fp -RRB- = tP -LRB- loop -RRB- = tP -LRB- fp -RRB- , and by normalizing q + p = 1 we compute : SP -LRB- fp -RRB- -LRB- n ; p -RRB- = tP -LRB- loop -RRB- tP -LRB- fp -RRB- = -LRB- 1 ? p -RRB- n + pnP -LRB- 1 ? p -RRB- + pnP = Pn -LRB- 1 ? p -RRB- + pn P -LRB- 1 ? p -RRB- + pn : -LRB- 5 -RRB- Figure 3 -LRB- a -RRB- is a plot of functional speedup for varying numbers of processors when n = 4 and p = -LRB- 0:5 ; :8 ; :9 -RRB- . Experimental points from a synthetic benchmark that implements the task graph of Figure 2 -LRB- a -RRB- are also shown . We now use the same task graph of Figure 3 -LRB- a -RRB- but we make the parallel part of task i be of size ip . The serial execution times for task Ti and the entire task graph are given by : tTi = q + ip ; tS = Pni = 1 -LRB- q + ip -RRB- = nq + pn -LRB- n +1 -RRB- 2 : -LRB- 6 -RRB- We can exploit loop and functional parallelism as before , dividing the P processors into static partitions of size P = n . The functional parallelism time is given by the time to execute the longest task and we obtain : tP -LRB- loop -RRB- = n X i = 1 -LRB- q + ip P -RRB- = nq + p P n -LRB- n + 1 -RRB- 2 ; -LRB- 7 -RRB- 4 8 12 16 20 24 28 32 0.5 1 1.5 2 2.5 3 3.5 4 Number of Processors Functional Speedup Functional Parallelism - Homogeneous Tasks p = 0.50 p = 0.80 p = 0.90 n = 4 -LRB- a -RRB- 4 8 12 16 20 24 28 32 0.5 1 1.5 2 2.5 3 3.5 4 Number of Processors Functional Speedup Static Functional Parallelism - Nonhomogeneous Tasks p = 0.50 p = 0.80 p = 0.90 n = 4 Static Partitions -LRB- b -RRB- 4 8 12 16 20 24 28 32 0.5 1 1.5 2 2.5 3 3.5 4 Number of Processors Functional Speedup Dynamic Functional Parallelism - Nonhomogeneous Tasks p = 0.50 p = 0.80 p = 0.90 n = 4 Dynamic Partitions -LRB- c -RRB- 4 8 12 16 20 24 28 32 0.25 0.5 0.75 1 1.25 1.5 1.75 2 Number of Processors Functional Speedup Internal Functional Parallelism n = 16 -LRB- d -RRB- Fig. 3 . Functional speedup as a function of the number of processors and fraction of parallelism p . In the above plots the lines represent results from the analytical models and the points are actual measurements of synthetic benchmarks . Parameter n is the degree of functional parallelism . tP -LRB- fps -RRB- = q + pn P = n = q + p P n2 ; -LRB- 8 -RRB- SP -LRB- fps -RRB- -LRB- n ; p -RRB- = tP -LRB- loop -RRB- tP -LRB- fps -RRB- = 2Pn -LRB- 1 ? p -RRB- + pn -LRB- n + 1 -RRB- 2 -LRB- P -LRB- 1 ? p -RRB- + pn2 -RRB- : -LRB- 9 -RRB- The functional speedup is less than 1 for q < pn 2P , in which case the exploitation of functional parallelism is detrimental to performance . The problem is the static partitioning of processors . Further performance improvements can be obtained by allowing dynamic assignment of processors to task . As the smaller tasks finish , their processors are reassigned to other tasks . With a uniform distribution of work among all P processors the parallel execution time and functional speedup are given by : tP -LRB- fpd -RRB- = q + p P n -LRB- n + 1 -RRB- 2 ; -LRB- 10 -RRB- SP -LRB- fpd -RRB- -LRB- n ; p -RRB- = tP -LRB- loop -RRB- tP -LRB- fpd -RRB- = 2Pn -LRB- 1 ? p -RRB- + pn -LRB- n + 1 -RRB- 2P -LRB- 1 ? p -RRB- + pn -LRB- n + 1 -RRB- : -LRB- 11 -RRB- SP -LRB- fpd -RRB- -LRB- n ; p -RRB- is always greater than 1 and greater than SP -LRB- fps -RRB- -LRB- n ; p -RRB- . Analytical curves and experimental points from a synthetic benchmark can be seen in Figures 3 -LRB- b -RRB- and 3 -LRB- c -RRB- for static and dynamic partitions , respectively . We now consider the parallel loop in Figure 2 -LRB- b -RRB- , which illustrates loop parallelism with internal functional parallelism . Let each task Ti be strictly serial with execution time t . The serial and loop parallel execution times of the loop nest are given by : tS = P -LRB- P +1 -RRB- 2 nt ; tP -LRB- loop -RRB- = P nt : -LRB- 12 -RRB- When the functional parallelism inside each loop iteration is exploited , processors that finish their iterations early become available to execute tasks of iterations from other processors . Ideally , the tasks are uniformly distributed among the processors . In this case the execution time and functional speedup are given by : tP -LRB- fp -RRB- = P -LRB- P + 1 -RRB- 2 n P t : -LRB- 13 -RRB- SP -LRB- fp -RRB- = tP -LRB- loop -RRB- tP -LRB- fp -RRB- = Pnt -LRB- P +1 -RRB- n 2 t = 2P P + 1 : -LRB- 14 -RRB- A plot of this speedup and experimental points for n = 16 are shown in Figure 3 -LRB- d -RRB- . 8 Description of Benchmarks In order to demonstrate the efficiency of nanoThreads in real programs , several benchmarks have been tested . The time measured is the time actually required to perform the requisite calculations , excluding the time required to initialize the system or read in data . We give here a quick description of each benchmark . Matrix Multiply -LRB- MM -RRB- : The matrix multiply application is written in C++ using the nanoThreads library . The code simply performs a straightforward implementation of the definition of matrix multiply -LRB- cij = P k aikbkj -RRB- . The two outermost loops are run in parallel as doall loops . Strassen 's Matrix Multiply -LRB- SMM -RRB- : An implementation of Strassen 's matrix multiply algorithm -LSB- 11 -RSB- was also written in C++ using the nanoThreads library . Strassen 's algorithm is a recursive algorithm that breaks down a matrix into four quadrants , recursively performs multiplies on the quadrants and performs a combination of matrix adds and subtracts . The corresponding task graph is shown in Figure 4 . Where a traditional matrix multiply of two n ? n matrices is O -LRB- n3 -RRB- , Strassen 's algorithm is O -LRB- n2 :807 -RRB- . Because the overhead is high , our implementation uses Strassen 's algorithm only for large -LRB- > = 64 ? 64 -RRB- matrices . When the quadrant size drops below 64 , then the traditional parallel matrix multiply is used . ComplexMatrix Multiply -LRB- CMM -RRB- : AC + + implementation using the nanoThreads library . A complex multiply is equivalent to four real multiplies and two real adds . As shown in the corresponding task graph of Figure 4 , these matrix operations represent functional parallelism . Each matrix A is represented by two matrices , one real and one imaginary , A = Ar + jAi . The matrix C = A ? B is calculated according to the equation : C = -LRB- ArBr ? AiBi -RRB- + j -LRB- ArBi + AiBr -RRB- : -LRB- 15 -RRB- All of the measurements with the matrix multiplies in this paper were taken with matrices of size 256 ? 256 . Two Dimensional Fast Fourier Transform -LRB- FFT2 -RRB- : Fortran code that performs a two dimensional fast Fourier transform was obtained fromCMU ` sTask Parallel Suite -LSB- 5 -RSB- and hand-parallelized with the C++ nanoThreads library . There are two nested parallel loops that are exploited . There is also one routine performing matrix transpose that exploits functional parallelism . Perfect Benchmark TRFD -LRB- TRFD -RRB- : AC + + version of the Perfect Clubbenchmark TRFD was implemented with the nanoThreads library . This code was written from a high-level description of the functionality of TRFD . The code operates on fourdimensional triangular matrices , performing several multiplies and rearrangements . Only three loops were parallelized . Two of the parallel loops are perfectly nested . Quicksort -LRB- QUICK -RRB- : Quicksort -LSB- 14 -RSB- is a recursive ? divide and conquer ? algorithm that sorts a vector of elements . It does so by partitioning the vector into two parts , based on the value of the elements , and recursively applying the same technique to both parts . In the average case , Quicksort is O -LRB- n log n -RRB- . Because of high fixed overhead , vectors below a certain size -LRB- 256 elements -RRB- are sorted using a conventional nonrecursive sorting technique . The two recursive calls are independent and parallelized using functional parallelism . This benchmark was coded in Cedar Fortran , and the autoscheduling compiler automatically generated C++ autoscheduling code . The measurements were taken with a vector of size 1048576 elements . Computational Fluid Dynamics -LRB- CFD -RRB- : This is the kernel of a Fourier-Chebyshev spectral computational fluid dynamics code -LSB- 15 -RSB- . The task graph representing this code is shown in Figure 4 . It consists of 4 stages that operate on matrices of size 128 ? 128 . The first stage involves six 2-dimensional FFTs , the second stage six element by element matrix multiplies , the third stage three matrix subtracts , and the fourth and final stage three 2-dimensional FFTs . The parallelism at the task graph level is exploited , as is the loop parallelism inside each matrix operation -LRB- ? ; ? ; FFT -RRB- . This benchmark was coded in Cedar Fortran , and translated by the autoscheduling compiler . 9 Results The results presented in this section are compiled from a set of execution times for each benchmark . The experimental points were obtained by running each program three times on a given number of processors and selecting the minimum execution time observed . For the nanoThreads library benchmarks , a distributed queue was used . For the autoscheduling compiler benchmarks , a centralized queue was used . In all cases , the granularity control parameter was set to 1 -LRB- see Section 5 -RRB- . 9.1 Measurements of Speedup Figure 5 contains speedup plots for the seven benchmarks . In each case the speedup SP on P processors is computed as the ratio of serial execution time -LRB- for a serial version of the benchmark -RRB- to the parallel execution time on P processors . Different versions of parallel code are tested for each benchmark . + + + + + * * * * + + + + * * + * + + + + + + + + STOP START F F F F F F F F F * * * * * * * * * * + - - - STOP - SMM CMM CFD START STOP START Fig. 4 . Task graphs for SMM , CMM , and CFD . F , * , + , and ? stand for 2D FFT , matrix multiply -LRB- element-by-element multiply in the case of CFD -RRB- , matrix add , and matrix subtract , respectively . For the real and complex matrix multiplies , MM and CMM , speedups are shown for the cases with -LRB- GC -RRB- and without -LRB- No GC -RRB- granularity control . Both benchmarks contain two nested parallel loops , thus each contains enough loop parallelism to occupy all available processors . In addition , CMM exploits an outer level of functional parallelism . The effect of granularity control is noticeable in reducing the overhead of exploiting too many loops in parallel . This benefit increases with the number of processors , since the overhead for fetching iterations from a parallel loop is proportional to the number of processors . For Strassen 's algorithm -LRB- SMM -RRB- and the two-dimensionalFFT -LRB- FFT2 -RRB- , four versions are compared : with functional parallelism and granularity control -LRB- FP and GC -RRB- , with functional parallelism but without granularity control -LRB- FP -RRB- , with granularity control but withoutfunctional parallelism -LRB- GC -RRB- , no granularity control and no functional parallelism -LRB- No FP or GC -RRB- . The exploitation of functional parallelism in Strassen 's algorithm is particularly important because it is a recursive algorithm and the loop parallelism occurs only at a smaller granularity . Because the recursive calls result in an exponential number of successively smaller tasks , granularity control has a significant effect as well . Only the version with functional parallelism and granularity control achieved reasonable speedup . The body of the innermost parallel loop of benchmark FFT2 is very small . As a result , granularity control has a significant effect on its performance since it avoids unnecessary small grain parallelization . With granularity control , the version without functional parallelism generally achieves better performance than the version that exploits functional parallelism . The only functional parallelism in FFT2 that was exploited is in a matrix transpose function which causes bad cache performance . By using a profiler , pixie , we found that the number of instructions performed by the FP version is less than the number of instructions performed by the No FP version , since less time is spent by the schedulers spin waiting while there is no work to do . This indicates that the performance in a perfect memory system would be better with functional parallelism . 4 8 12 16 2 4 6 8 10 12 Number of Processors Speedup Speedup of MM GC No GC 4 8 12 16 20 24 28 2 4 6 8 10 12 Number of Processors Speedup Speedup of CMM GC No GC 4 8 12 16 1 2 3 4 5 6 Number of Processors Speedup Speedup of SMM FP and GC FP GC No FP or GC 4 8 12 16 20 24 28 1 2 3 Number of Processors Speedup Speedup of FFT2 FP and GC FP GC No FP or GC 4 8 12 16 20 24 28 2 4 6 8 10 12 14 16 18 20 Number of Processors Speedup Speedup of TRFD GC No GC SGI 2 4 6 8 10 12 2 4 6 8 Number of Processors Speedup Speedup of QUICK GC No GC 2 4 6 8 10 12 2 4 6 Number of Processors Speedup Speedup of CFD FP FP and LP LP SGI Fig. 5 . Speedup plots for the various benchmarks . Speedups of TRFD are shown for the cases of nanoThreads with granularity control -LRB- GC -RRB- , nanoThreads without granularity control -LRB- No GC -RRB- , and parallel loop constructs provided by the SGI compiler -LRB- SGI -RRB- . This benchmark has no functional parallelism , but still nanoThreads performs better than the SGI parallel loops . The granularity control version does not perform as well as the version without this control . This is a result of poor load balance resulting from the inhibited parallelism due to granularity control . This is an example of the case in which the smallest tasks are much larger than the scheduling overhead . TRFD has a limited amount of large grain parallelism and , therefore , the cost of unrestricted parallelism -LRB- no granularity control -RRB- is negligible . On the other hand , granularity control restricts some of the parallelism , in detriment of load balance . The SGI compiler restricts parallelism even more since it does not exploit nested parallelism , thus further hampering load balance . Quicksort -LRB- QUICK -RRB- only has functional parallelism . Versions with -LRB- GC -RRB- and without -LRB- No GC -RRB- granularity control are compared . We observe a 75 % improvement on the speedup for 12 processors when granularity control is used . As with Strassen 's algorithm , granularity control here avoids an exponential explosion of tasks . For CFD we compared four versions : autoscheduling compiler with loop parallelism only -LRB- LP -RRB- , autoscheduling compiler with functional parallelism only -LRB- FP -RRB- , autoscheduling compiler with loop and functional parallelism -LRB- LP and FP -RRB- , SGI compiler with loop parallelism only -LRB- SGI -RRB- . We note that the best performance was obtained by exploiting functional parallelism only . This is mainly the result of a better cache behavior in the functional parallel version . Each task operates on entire independent matrices and therefore there is little cache interference between processors . In the loop parallel versions , processor operate concurrently on the same matrices . The SGI compiler was better than the autoscheduling compiler in exploiting loop parallelism . 9.2 Improvements due to Granularity Control Figure 6 lists the improvement in execution time as a function of the number of processors when granularity control is used for six of the benchmarks . Let tPG be the execution time on P processors with granularity control and tPN be the execution time on P processors without granularity control . The percentage improvement for P processors is defined as : I % P = tPN ? tPG tPN ? 100 -LRB- 16 -RRB- The effect of granularity control varies significantly depending on the characteristics of the algorithm and the size of the smallest task . Granularity control can make a large difference in performance , especially when the available parallelism is not large -LRB- FFT2 , SMM -RRB- . In the case where granularity control performs worse -LRB- TRFD -RRB- , the difference is small . 10 Related Work A variety of thread management issues and implementation alternatives are considered in -LSB- 1 -RSB- . It includes analysis and implementation results for different types of queue management and lock synchronization techniques . 80706050403020100-10 % Improvement AAAAAA AA AA AA AAAA AA AA AA AAAA AA AA AA AAAA AA AA AA AA AAAA AA AA AA AA AA AA AA AAA AAA AA AA AA AAAA AA AA AAAA AA AA AA Number of Processors124681216202428P ercentage Improvements of Granularity ControlC MMMMFFTTRFDSMM AA AA Quicksort Percentage Improvement Due To Granularity Control Number of processors 1 2 4 6 8 12 16 20 24 28 CMM 19.6 29.0 26.7 19.6 39.5 25.0 28.2 17.2 26.3 8.2 MM -5.2 3.3 7.9 12.8 9.8 14.0 12.4 ? ? ? FFT2 45.7 75.2 78.1 75.7 76.5 78.2 76.0 74.6 71.4 64.3 TRFD 4.6 -0.1 -3.6 5.1 3.2 3.5 -9.1 0.0 -2.0 -9.2 SMM 19.2 31.7 43.9 52.2 59.5 65.5 61.0 ? ? ? Quicksort 6.96 15.04 24.64 29.39 36.11 41.79 ? ? ? ? Fig. 6 . The effect of granularity control . Scheduler Activations -LSB- 2 -RSB- and Process Control -LSB- 12 -RSB- both offer user-level scheduling and dynamic adaptation to changing environments . Scheduler activations supply an execution context that may be manipulated by a user-level scheduler or the kernel to allow for user-level scheduling with the flexibility and power of kernel-level threads . Allowed kernel manipulations include the grantingor removing of scheduler activations . The process control approach allows the user to request processors and gives the user control over when to release processors . Processors are released by the user on request of the kernel at safe points to prevent deadlock and similar problems . Considerable work has been done on the exploitation of functional and loop parallelism by the Paradigm -LSB- 20 -RSB- group at the University of Illinois . They rely on a Macro Dataflow Graph which is a directed acyclic task graph in which the nodes are weighted with the communication and computation time . Their approach performs static partitioning for data and computations on a distributed system to minimize total runtime . Jade -LSB- 21 -RSB- is a high-level language designed for the exploitation of coarse-grain task -LRB- functional -RRB- parallelism . Concurrency is detected dynamically from data access specifications in each task . The language also supports the declaration of hierarchical tasks . Techniques to exploit nested parallelism include switch-stacks -LSB- 4 -RSB- and process con - trol blocks -LRB- PCBs -RRB- -LSB- 13 -RSB- . A PCB for a parallel loop is used to schedule the iterations of that loop . Reference -LSB- 13 -RSB- discusses heuristics that strike a balance between efficient allocation of PCBs versus load balancing problems that arise from barrier synchronization in nested parallel loops . Switch-stacks handle nested parallelism by actually swapping stacks between processors so no one processor is left idle waiting for another to finish . The Psyche -LSB- 16 -RSB- system has facilities for user-level threads in which many tasks normally performed by the kernel , such as interrupt handling and preemptions , are handled at the user-level . Like nanoThreads , it relies on multiple virtual processors sharing the same address space . Chores -LSB- 6 -RSB- is a paradigm for the exploitation of loop and functional parallelism . It allows dynamic scheduling at the user-level and the expression of dependences between tasks without explicit synchronization . 11 Conclusion In this paper we have demonstrated the benefits of exploiting functional parallelism and performing dynamic granularity control . Our implementation is based on a user-level threads model that performs dynamic task scheduling . In particular , we have shown that the exploitation of nested functional parallelism is beneficial both in terms of increasing the amount of high-level parallelism andimproving the load balance of parallel loops . We have also shown the potential for significant improvements by dynamically controlling the granularity of exploited parallelism . Results from synthetic benchmarks were used to verify analytical models of performance . Measurements with application kernels demonstrated the efficiency of our approach for real scientific computations on an existing commercial shared-memory multiprocessor . References 1 . Thomas Anderson , Edward Lazowska , and Henry Levy . The performance implications of thread management alternatives for shared-memory multiprocessors . IEEE Transactions on Computers , 38 -LRB- 12 -RRB- , December 1989 . 2 . Thomas E. Anderson , Brian N. Bershad , Edward D. Lazowska , and Henry M. Levy . Scheduler activations : Effective kernel support for the user-level management of parallelism . In 13th ACM Symposium on Operating Systems Principles , pages 95 ? 109 . ACM Sigops , October 1991 . 3 . Carl J. Beckmann . Hardware and Software for Functional and Fine Grain Parallelism . PhD thesis , Department of Electrical and Computer Engineering , University of Illinois at Urbana-Champaign , 1993 . 4 . Jyh-HerngChow andWilliams Ludwell Harrison . Switch-stacks : A schemefor microtasking nested parallel loops . In Supercomputing 90 , pages 190 ? 199 , Nov. 1990 . 5 . Peter Dinda , Thomas Gross , David O'Hallaron , Edward Segall , James Stichnoth , Jaspal Subhlok , Jon Webb , and Bwolen Yang . The CMU task parallel program suite . Technical Report CMU-CS-94-131 , School of Computer Science , Carnegie-Mellon University , March 1994 . 6 . Derek Eager and John Zahorjan . Chores : Enhanced run-time support for shared-memory parallel computing . ACM Transactions on Computer Systems , 11 -LRB- 1 -RRB- , February 1993 . 7 . Mike Galles and Eric Williams . Performance optimizations , implementation , and verification of the SGI Challenge multiprocessor . Silicon Graphics Technical Report . Available from http://www.sgi.com . 8 . M. Girkar and C. D. Polychronopoulos . The HTG : An intermediate representation for programs based on control and data dependences . Technical Report 1046 , Center for Supercomputing Research and Development , University of Illinois at Urbana-Champaign , May 1991 . 9 . Milind Girkar . Functional Parallelism : Theoretical Foundations and Implementation . PhD thesis , Department of Computer Science , University of Illinois at Urbana-Champaign , 1992 . 10 . Milind Girkar and Constantine Polychronopoulos . Automatic detection and generation of unstructured parallelism in ordinary programs . IEEE Transactions on Parallel and Distributed Systems , 3 -LRB- 2 -RRB- , April 1992 . 11 . Gene H. Golub and Charles F. Van Loan . Matrix Computations . The Johns Hopkins University Press , Baltimore , MD , 1989 . 12 . Anoop Gupta , Andrew Tucker , and Luis Stevens . Making effective use of shared-memory multiprocessors : The process control approach . Technical Report CSL-TR-91-475A , Computer Systems Laboratory , Stanford University , 1991 . 13 . S. F. Hummel and E. Schonberg . Low-overhead scheduling of nested parallelism . IBM J. Res . Develp. , 35 -LRB- 5/6 -RRB- :743 ? 765 , Sept/Nov 1991 . 14 . D. E. Knuth . The Art of Computer Programming , Vol . 3 Sorting and Searching . AddisonWesley , Reading , Mass. , 1973 . 15 . S. L. Lyons , T. J. Hanratty , and J. B. MacLaughlin . Large-scale computer simulation of fully developed channel flow with heat transfer . International Journal of Numerical Methods for Fluids , 13:999 ? 1028 , 1991 . 16 . Brian D. Marsh , Michael L. Scott , Thomas J. LeBlanc , and Evangelos P. Markatos . Firstclass user-level threads . In Proceedings of the 13th ACM Symposium on Operating Systems Principles , pages 110 ? 121 , October 1991 . 17 . C. D. Polychronopoulos , M. B. Girkar , Mohammad R. Haghighat , C. L. Lee , B. Leung , and D. A. Schouten . Parafrase-2 : An environment for parallelizing , partitioning , synchronizing , and scheduling programs on multiprocessors . International Journal of High Speed Computing , 1 -LRB- 1 -RRB- :45 ? 72 , May 1989 . 18 . Constantine Polychronopoulos , Nawaf Bitar , and Steve Kleiman . nanothreads : A user-level threads architecture . In Proceedings of the ACM Symposium on Principles of Operating Systems , 1993 . 19 . Constantine D. Polychronopoulos . Autoscheduling : Control flow and data flow come together . Technical Report 1058 , Center for Supercomputing Research and Development , University of Illinois at Urbana-Champaign , 1990 . 20 . Shankar Ramaswamy and Prithviraj Banerjee . Processor allocation and scheduling of macro dataflow graphs on distributed memory multicomputers by the PARADIGM compiler . In International Conference on Parallel Processing , pages II :134 ? 138 , St. Charles , IL , August 1993 . 21 . Martin C. Rinard , Daniel J. Scales , and Monica S. Lam . Jade : A high-level machineindependent language for parallel programming . IEEE Computer , 26 -LRB- 6 -RRB- :28 ? 38 , June 1993 . This article was processed using the LATEX macro package with LLNCS style

O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

dynamic scheduling
functional parallelism
parallel processing
task granularity
threads


T9000 - A Preliminary Evaluation of Arithmetic Performance G. Bader ? and B. Przywara Interdisziplin?ares Zentrum f?ur Wissenschaftliches Rechnen Universit?at Heidelberg Im Neuenheimer Feld 368 D-6900 Heidelberg t9@iwr1.iwr.Uni-Heidelberg.de May 5 , 1993 Abstract This brief note presents a first assessment of the arithmetic performance of the new transputer processors T9000 . The presented results are obtained on a T9000 version of the processor running with reduced clock speed of 10 MHz . Due to the still reduced functionality of the available processor we consider only computational kernels of BLAS type . Nevertheless , we hope that these preliminary tests provide some idea of the computing power of the T9000 processor for real applications . ? present address : Institut f?ur Mathematik , Technische Universit?at Cottbus , Karl-Marx-Str . 17 , O-7500 Cottbus 1 Introduction Distributed memory parallel computers based on transputers of the T800 family have been available for 8 years now . They have found - mostly in the academic community - high acceptance . The success of this technology can be contributed to a large extend to the favorable balance between computational performance and communication speed . This allows in particular the efficient use of such systems with large numbers of processors and the implementation of algorithms with fine granularity . Meanwhile T800 systems do no longer satisfy the need for continuously growing computing power . The new processor T9000 has been announced to provide an approximately 10 fold increase in computing power and , what is equally important , communication bandwidth . This has let , long before the silicon was available , to expectations about the actual performance of the T9000 processor for practical applications . With this note we hope to provide an early assessment of the potential performance of this new transputer processor . The now available version T9000 of the processor has still limited functionality and is running with a reduced clock speed of only 10 MHz . At present it is able to reach a peak performance of 40 Mips and 3.33 MFlops . The final version of the T9000 transputer was announced to run with a clock speed of 50 MHz . It should provide a peak performance of 200 Mips and 25 MFlops . The crucial question for advanced processor technologies is , to what extend this peak performance can be put to work for practical applications . The present paper documents some performance tests done for the the version of the T9000 processor . We hope that these results are a contribution towards the assessment of this processor . We also try to provide , on the basis of measured results , an estimate for the performance of the final 50 MHz version of the processor . 2 Microprocessor architecture The main emphasis of this section is to provide a brief overview of the architecture of the T9000 processor as far as they are relevant for understanding the performance tests carried out . A presentation of the details can be found in a number of technical publications , see for example -LSB- 3 , 4 , 5 -RSB- . We follow closely the presentations -LSB- 6 , 10 -RSB- . The T9000 comprises a superscalar 32-bit processor -LRB- CPU -RRB- with a 64-bit floating point unit -LRB- FPU -RRB- , a communication processor -LRB- VCP -RRB- together with 4 communication links , a control unit and its associated links , an external memory interface -LRB- EMI -RRB- and 16 KBytes of on-chip memory -LRB- Cache -RRB- . The final component of the T9000 is a crossbar switch -LRB- X-Bar -RRB- which connects the other components . The T9000 operates at 50 MHz approximately 10 times as fast as a T805 processor with 30 MHz . Thus , it issues approximately 4 times as many memory accesses per cycle as the T805 . The 4 communication links provide a maximum bandwidth of 70 MBytes/sec as compared to 11.2 MBytes/sec for the T805 . This implies that the T9000 communication system makes 2.5 times as many data accesses per cycle . In order to reduce the requirements in terms of bandwidth on the memory system the T9000 uses two levels of caching . The first level is within the processor . It provides caching of data in the workspace of the currently executed process with a capacity of 32 words . As a high percentage of access made by the processor is for data in the workspace this has a significant effect . The second level of caching is located between the major functional units and the external memory . Ultimately , however , access must be made to external memory . The memory interface of the T9000 provides a 64-bit wide memory access , giving a maximum of 200 MByte/sec . The T9000 has 16 KBytes of on-chip memory . This RAM can be configured as 16 or 8 KBytes of cache . The latter allows to lock code and data into fast on-chip memory for time critical processes . The cache allows that external memory accesses are mostly made for complete cache lines . They are chosen to be 4 words long , e.g. 4 single or 2 double precision data items . This reduces effectively the requirements for external memory bandwidth . Note , however , that the processor can also be used without external memory . It operates then directly on the fast on-chip memory without cache . Even if caching reduces the requirements for external memory bandwidth there remains the necessity to provide processor and VCP with sufficient bandwidth to the cache . The internal memory is divided into 4 banks , each caching a quarter of the address space . When an address is presented to the memory system the request is routed through the crossbar to the appropriate memory bank . There exist 9 ports onto the crossbar , 1 for the EMI , 4 for the processor , 3 for the VCP and 1 shared between the scheduler and the control unit . Each bank of the cache can provide access to one word every cycle and thus the total on-chip memory bandwidth is 800 MByte/sec . Figure 1 provides an overview of the main building blocks of the processor and their interconnection . instructionbuffer8 wordsCPUi nstruction fetch2 non-local read ports2 localr ead portswriteport work space cache -LRB- write-through -RRB- 32 wordsmain memorym ain cache -LRB- write-back -RRB- 4 K wordsnon-cachedreads -LRB- 32 bit -RRB- c achedreads32/64 bitc achedwrites32/64 bitn on-c achedwrites -LRB- 32 bit -RRB- Figure 1 : T9000 processor architecture The T9000 executes most of the instructions in a single cycle . This is in contrast to T800 transputers . Especially most of the slower instructions of the T800 have been significantly improved : cycles T800 T9000 fpadd 7 2 -LRB- double precision , 64 bit -RRB- fpmul 20 3 -LRB- double precision , 64 bit -RRB- mul 36 2 ? 5 shl n n 1 Already the T800 processor allows for the integer and floating point units to operate in parallel . This is known as a superscalar architecture . The T9000 processor takes this kind of concurrency much further . Important classes of instructions , such as address calculations , the loading of integer or floating point numbers , the operations on these and the final storing are done in separate processor units . Actually , some of the most frequent operations are done on multiple functional units : stages instruction class units instructions 1 local memory read 2 e.g. ldl , ldlp , ldc , : : : 2 address calculations 2 e.g. ldnlp , wsub , : : : 3 non-local memory read 2 e.g. ldnlp , wsub , : : : 4 int or fpu operations 1 e.g. add , fpadd , : : : 5 memory write 1 e.g. stl , stnl , fpstnl , : : : 6 jump 1 e.g. eqc , cj , j , : : : AA AA AA AA AA fetch1 word/cycle ? Group8 bytes/cycleLocal2 work-spacecachereads/cycle Address2 non-localaddresses/cycle -- > 2 readsofread + write Read2 maincachereads/cycle Function1 integeror floatingpointoperation/cycleWrite1 writeto cache/cycle AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAA AAAAA AA AA AA AA AA AA AA AAAAA AAA AAA AAA AAA AAA AAA AAA AAAAA AA AA AA AAA AAA AAA AAA AAA AAA AAA AAA Jump Figure 2 : T9000 processor pipeline The processor units are organized as a pipeline , e.g. figure 2 . In order to maximize the throughput of the processor pipeline , the processor is equipped with a hardware implemented grouping mechanism . It takes the incoming stream of instructions , builds groups of instructions , and feeds those into the pipeline with a rate of one group per cycle . 3 Evaluation of computational kernels In order to make the grouping mechanism and the implications for arithmetic kernels more transparent we consider two examples of BLAS Level 1 routines in detail . See -LSB- 8 -RSB- for the definition of these Basic Linear Algebra Subroutines . The first example is the calculation of scalar products . We limit our considerations to the single and double precision versions of sdot and ddot respectively . The actual coding of these routines is done with loop unrolling to reduce the loop overhead . Table 1 presents a typical assembly language segment for sdot . Note that the rules in table 1 indicate the groups formed in the processor pipeline . For the corresponding double precision version ddot the single precision assembler instructions fpldnlsn and fpldnlmulsn have to be replaced by its double precision instructions fpldnldb and fpldnlmuldb , respectively . 1 ldl x 6 instructions 2 ldnlp 2 8 bytes 3 fpldnlsn 4 ldl y 5 ldnlp 2 6 fpldnlmulsn 1 fpadd 1 instruction 2 bytes Table 1 : assembler sequence sdot Based on this assembler sequence we find that 6 processor cycles are necessary within the pipeline to evaluate the two groups on the T9000 . On the available 10 MHz version of the processor we can reach 3.33 MFlops . This is actually the peak performance of this processor , c.f. -LSB- 6 -RSB- . Note that this estimate is based on the evaluation of many sequences of the form given in table 1 . Hence , it is assumed that the pipelining effect leads to a partial overlapping of the sequences . For this reason we call such estimates theoretical performance . It does not contain any cycles necessary for loop overhead and for potential bank conflicts which are likely to occur when loading the data . Thus , the measured performance results are bound to be slightly lower than these numbers suggest . For the corresponding double precision routine ddot we find 7 cycles for the evaluation of the two groups and 2.86 MFlops of theoretical performance . The second example is the BLAS routine saxpy . A typical assembler language sequence is provided in table 2 . For the single precision routine saxpy we need 6 cycles for the evaluation of the two groups of table 2 giving a theoretical performance of 3.33 MFlops . Note , that saxpy needs 2 load and 1 store operations more than sdot . This leads inevitably to memory bank conflicts . Consequently , the measured results , c.f. table 3 , are bound to be lower than for sdot . 1 ldlp a 5 instructions 2 fpldnlsn 7 bytes 3 ldl x 4 ldnlp 2 5 fpldnlmulsn 1 ldl y 6 instruction 2 ldnlp 2 8 bytes 3 fpldnladdsn 4 ldl y 5 ldnlp 2 6 fpstnlsn Table 2 : assembler sequence saxpy The double precision version needs 8 cycles to evaluate and gives a theoretical performance of 2.5 MFlops . For this routine we have to load 3 and store 1 double precision data . This takes 8 processor cycles for data movement within the processor . The actual computation , 1 multiply and 1 add , takes only 5 cycles for completion . Thus , in this case a saturation of the internal memory bandwidth takes place . The limiting factor for the performance is data movement and not computational speed . Similar effects show for a number of double precision BLAS level 1 routines . Note , however , that this is also known for other superscalar architectures and in particular for vector processors . The situation improves for BLAS level 2 and 3 routines . 4 Measured performance results Before presenting the results obtained we outline briefly the available test environment . The hardware consists of a Parsytec GC-cube with a test board and a single T9000 processor . The test board also carries a T805 processor . It is used to load the T9000 processor with executable code and data via its dual ported RAM . The T9000 processor is running under a reduced clock speed of 10 MHz . At present the processor has a peak performance of 40 Mips and 3.33 MFlops . Finally , the board is equipped with 4 MByte of external memory . The software environment at the present state is the INMOS C-compiler , -LSB- 2 -RSB- , and a preliminary PARIX-T9000 version from Parsytec . The PARIX environment is called PHOENIX , see -LSB- 9 -RSB- . It allows to work with FORTRAN and C. Both systems are so called cross systems , e.g. the source code is compiled on the host computer . The executable code is loaded onto the transputer and executed by a run time system . 4.1 On-chip results As pointed out in section 2 , the T9000 can run without external memory . In this mode of operation the cache is not needed . This implies that the 16 KBytes of on-chip RAM are accessed without using the cache . As not all functions of the cache are yet fully operational we have chosen to evaluate the processor performance in this mode . The presented results should give an good idea of what percentage of the 3.33 MFlops of peak performance show up in real computations . We have tested a selected number of BLAS level 1 routines with the IN - MOS C-compiler . Both source code -LRB- src -RRB- and optimized assembler -LRB- asm -RRB- coding are considered . The obtained results are compiled in table 3 . Depending on the various routines , the results show good performance . This is in particular true for almost all single precision routines . For some double precision routines the limitation by the internal memory bandwidth can be observed . function MFlops src asm daxpy 1:633 1:896 saxpy 2:254 3:030 ddot 2:353 2:694 sdot 2:623 3:101 dscal 1:127 1:481 sscal 2:116 2:367 idamax :279 1:227 isamax :366 1:365 Table 3 : Performance results T9000 Now consider the Gauss-Seidel iteration for a standard five point discretization of the Laplace equation in 2 dimensions . The algorithm may serves as a simple model for computational fluid dynamic problems . Recall that for a N by N regular grid this algorithm solves iteratively a linear system of dimension N ? N with a very sparse coefficient matrix . In terms of processing this implies that large arrays of data are recursively touched while only a moderate amount of computations is done . Thus , similar to BLAS level 1 routines , the effective bandwidth for data movement within the processor plays an important role for the observed performance . A typical result for the T9000 is shown in table 4 . Note , for comparison , the T800 processor running at 25 MHz performs for this algorithm with :556 MFlops . Gauss-Seidel MFlops src asm N = 16 1:150 1:586 Table 4 : Performance for Gauss-Seidel iteration 4.2 Results with cache and external memory In any large size real computations the data have to be stored in external memory . Thus , it is of prime importance to test the behavior of the processor under such real life assumptions . Any external memory system has typically less bandwidth than internal memory . Hence , we have to expect a certain reduction of the processor performance when used with external memory . The question is to what extend this reduction shows up . The answer is obviously problem dependent . In this respect BLAS level 1 routines are a difficult test for any processor with an internal cache . function MFlops no Cache Cache dsum :900 1:424 ssum 1:804 2:378 dasum :726 1:421 sasum 1:329 2:273 daxpy :601 :770 saxpy 1:629 2:045 ddot :824 1:225 sdot 1:521 2:040 Table 5 : Performance results with caching As the cache on the present version of the T9000 is not fully functional we have done only a preliminary testing so far . The results have to be interpreted with caution as it is not always clear to what extend the limited performance is due to the reduced functionality of the cache . In particular , the port between cache and external memory is only 32-bit wide . In the final version of the processor this will be extended to a 64-bit wide port . This yields a doubling of the corresponding bandwidth . In order to be able to compare the results with and without cache we have selected similar routines as in subsection 3.1 . The obtained results are presented in table 5 . When using external memory together with the cache the performance is considerably lower than the on-chip results , c.f. table 3 . This is , to a large extend , due do the presently limited external memory bandwidth . In general , however , it is difficult to give a reliable judgment of these results at present . This has to wait till all components of the processor operate as they should . Finally , we list the results for the Gauss-Seidel iteration . Iterative techniques are typically applied for large systems . Hence , the effectiveness of data movement between processor and external memory is of very high importance . The corresponding results are shown in table 6 . Gauss-Seidel MFlops no Cache Cache N = 16 :544 1:396 Table 6 : Performance for Gauss-Seidel iteration Summary Here we would like to give a brief summary and judgment of our , still limited , experiences with the new Transputer processor T9000 . Many more tests have to be carried through before the performance of this processor as a building block for massively parallel systems can be reliably evaluated . However , before this can be done the processor has yet to reach its full functionality . In particular , very little can be said at present on the processors balance between communication speed and computational performance . Nevertheless , some conclusions can be drawn from our testing . Lets first discuss the performance when the processor is running without external memory , i.e. without cache . The presented results prove that especially for single precision calculations the achieved performance is very close to the peak performance of the processor . For double precision data the results are , depending on the algorithm , to some extend limited by the internal memory bandwidth . Still , the observed performance is reasonable close to the peak performance of the processor . On the basis of the measured performance results a linear speedup can be expected when the processor is running with higher clock speed . Thus , we guess a five fold increase of the presented on-chip numbers for the 50 MHz version of the T9000 . In comparison to the T805 this would be a factor 8 to 10 of performance gain . When considering the results with cache and data stored in external memory the present situation is not quite as satisfactory . Especially , the presently limited cache to external memory bandwidth reduces the observed performance considerably . This should , however , improve substantially as soon as the memory interface becomes 64-bit wide . This and the not yet full functionality of the cache makes it more difficult to extrapolate the observed results for the 50 MHz version of the processor . However , applying a rather conservative estimate , it is reasonable to expect a performance gain of a factor 5 to 8 for the 50 MHz T9000 when compared to the T805 processor . On the same basis one might safely guess the absolute performance in the range between 6 and 10 MFlops for double precision computations . Finally , it should be mentioned that the superscalar architecture of the T9000 processor is less sensitive to the optimization quality of code produced by the compiler than the T800 family of processors . The performance reduction involved for less careful coded algorithms is rather moderate . On the other hand , writing an optimizing compiler might not be an easy task . References -LSB- 1 -RSB- G. Bader and B. Przywara , Arbeitsumgebungen und Leistung von Transputersystemen , in : Bader/Rannacher/Wittum eds. , Numerische Algorithmen auf Transputer-Systemen , Teubner Skripten zur Numerik , 1992 . -LSB- 2 -RSB- INMOS , ANSI C Toolset User Manual , Inmos 1990 . -LSB- 3 -RSB- INMOS : The T9000 Transputer Instruction Set Manual , Preliminary Information , Inmos 1992 -LSB- 4 -RSB- INMOS : The T9000 Transputer Hardware Reference Manual , Preliminary Information , Inmos 1992 -LSB- 5 -RSB- INMOS : The T9000 Transputer Products Overview Manual , INMOS Databook Series , Inmos 1991 . -LSB- 6 -RSB- F. L?ucking , T9000 : Microprocessor Architecture , Parallel Link , Special Issue Launch T9/CeBIT , Parsytec , 1993 . -LSB- 7 -RSB- F. L?ucking : Efficient BLAS level 1 Library for Transputers , Parsytec GmbH , Aachen 1990 . -LSB- 8 -RSB- C. Lawson , R. Hansen , D. Kincaid and F. Krough : Basic linear algebra subprograms for FORTRAN usage , ACM Trans . Math . Software 5 , 308 - 1371 , 1979 . -LSB- 9 -RSB- Parsytec Eastern Europe , PHOENIX PARIX , Release 0.9 , Chemnitz 1993 . -LSB- 10 -RSB- R. Shepherd : Next Generation Transputers and Beyond - 2 , T9000 : superscalar transputer , Aug. 1991 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O