Patent Publication Number: US-2016224502-A1

Title: Synchronization in a Computing System with Multi-Core Processing Devices

Description:
FIELD OF THE DISCLOSURE 
     The invention relates to synchronization within a computing system that contains a plurality of multi-core processing devices, and, in particular, synchronized processing of multiple computing resources of the multi-core processing devices by virtue of signaling of events, status, and/or activity related to buffers used within the multi-core processing devices to accommodate communication. 
     BACKGROUND 
     Information-processing systems are computing systems that process electronic and/or digital information. Typical information-processing system may include multiple processing elements, such as multiple single core computer processors or one or more multi-core computer processors capable of concurrent and/or independent operation. Such systems may be referred to as multi-processor or multi-core processing systems. 
     Synchronization mechanisms in such systems commonly include interrupts and/or exceptions implemented in hardware, software, and/or combinations thereof. When multiple processing elements, such as multiple processors or multiple processing cores execute in parallel to process data for one computation process, the interrupts and/or exceptions do not provide adequate synchronization between the processing elements. Therefore, there is a need in the art for a synchronization mechanism for a plurality of processing elements of a computing system that may be able to detect when a prescribed set of operations is complete and the system has become idle, that is independent of the number of operations involved and/or the specific length of time taken by each of those operations. 
     SUMMARY 
     The present disclosure provides systems, methods and apparatuses for synchronization of processing elements in a computing system. In one aspect of the disclosure, a processing device may be provided. The processing device may comprise a plurality of processing elements each configured to generate events, a plurality of buffers for communicating data to and from the plurality of processing elements, at least one programmable register to hold a predefined time limit, at least one timing register for counting a time since a last activity in one or more buffers, and at least one event register to hold an event flag. The event flag may be set to a signaled state to signal that an event has taken place when the time counted in the at least one timing register reaches the predefined time limit. 
     In another aspect of the disclosure, a method of operating a processing device that has a plurality processing elements configured to support parallel processing may be provided. The method may comprise loading one or more tasks to be executed in two or more processing elements of the plurality of processing elements, executing one or more tasks on the two or more processing elements and monitoring buffers associated with the two or more processing elements. The monitored buffers may be used to communicate the one or more tasks to the two or more processing elements. The method may further comprise determining states of the two or more processing elements based on the monitored buffer activities and setting a first event flag after no activity is monitored in at least one of the two or more processing elements based on the determined states. 
     In yet another aspect of the disclosure, a computing system may be provided. The computing system may comprise a plurality of processing device and a host. Each processing device may comprise a plurality of processing elements each configured to generate events, a plurality of buffers for communicating data to and from the plurality of processing elements, at least one programmable register to hold a predefined time limit, at least one timing register for counting a time since a last activity in one or more buffers and at least one event register to hold an event flag. The event flag may be set to a signaled state to signal that an event has taken place when the time counted in the at least one timing register reaches the predefined time limit. The host may be configured to assign one or more tasks to at least a subset of processing elements of the plurality of processing devices, load to one or more tasks to the assigned processing elements to be executed thereon, monitor event flag(s) associated with the assigned processing elements and determine whether one or more processing devices of the plurality of processing devices have entered an idle state. 
     In yet another aspect, the present disclosure may provide a method of operating a computing system that may have a plurality of processing devices and each processing device may have a plurality of processing elements configured to support parallel processing. The method may comprise assigning one or more tasks to at least a subset of processing elements of the plurality of processing devices, loading one or more tasks to the assigned processing elements, executing the one or more tasks on the assigned processing elements and monitoring buffers associated with the assigned processing elements. The monitored buffers may be used to communicate the one or more tasks to the two or more processing elements. The method may further comprise determining states of the assigned processing elements based on the monitored buffer activities and setting a first event flag after no activity is monitored in at least one of the two or more processing elements based on the determined states. 
     These and other objects, features, and characteristics of the present invention, as well as the methods of operation and functions of the related elements of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the invention. As used in the specification and in the claims, the singular form of “a”, “an”, and “the” include plural referents unless the context clearly dictates otherwise. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1A  is a block diagram of an exemplary computing system according to the present disclosure. 
         FIG. 1B  is a block diagram of an exemplary processing device according to the present disclosure. 
         FIG. 2A  is a block diagram of topology of connections of an exemplary computing system according to the present disclosure. 
         FIG. 2B  is a block diagram of topology of connections of another exemplary computing system according to the present disclosure. 
         FIG. 3A  is a block diagram of an exemplary cluster according to the present disclosure. 
         FIG. 3B  is a block diagram of an exemplary super cluster according to the present disclosure. 
         FIG. 4  is a block diagram of an exemplary processing engine according to the present disclosure. 
         FIG. 5  is a block diagram of an exemplary packet according to the present disclosure. 
         FIG. 6  is a flow diagram showing an exemplary process of addressing a computing resource using a packet according to the present disclosure. 
         FIG. 7  is a block diagram of an exemplary processing device according to the present disclosure. 
         FIG. 8  is a block diagram for an exemplary cluster according to the present disclosure. 
         FIG. 9  illustrates a computing system configured to synchronize processing elements according to the present disclosure. 
         FIGS. 10-11  illustrate methods for synchronizing processing engines according to the present disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     Certain illustrative aspects of the systems, apparatuses, and methods according to the present invention are described herein in connection with the following description and the accompanying figures. These aspects are indicative, however, of but a few of the various ways in which the principles of the invention may be employed and the present invention is intended to include all such aspects and their equivalents. Other advantages and novel features of the invention may become apparent from the following detailed description when considered in conjunction with the figures. 
     In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. In other instances, well known structures, interfaces, and processes have not been shown in detail in order to avoid unnecessarily obscuring the invention. However, it will be apparent to one of ordinary skill in the art that those specific details disclosed herein need not be used to practice the invention and do not represent a limitation on the scope of the invention, except as recited in the claims. It is intended that no part of this specification be construed to effect a disavowal of any part of the full scope of the invention. Although certain embodiments of the present disclosure are described, these embodiments likewise are not intended to limit the full scope of the invention. 
       FIG. 1A  shows an exemplary computing system  100  according to the present disclosure. The computing system  100  may comprise at least one processing device  102 . A typical computing system  100 , however, may comprise a plurality of processing devices  102 . Each processing device  102 , which may also be referred to as device  102 , may comprise a router  104 , a device controller  106 , a plurality of high speed interfaces  108  and a plurality of clusters  110 . The router  104  may also be referred to as a top level router or a level one router. Each cluster  110  may comprise a plurality of processing engines to provide computational capabilities for the computing system  100 . The high speed interfaces  108  may comprise communication ports to communicate data outside of the device  102 , for example, to other devices  102  of the computing system  100  and/or interfaces to other computing systems. Unless specifically expressed otherwise, data as used herein may refer to both program code and pieces of information upon which the program code operates. 
     In some implementations, the processing device  102  may include 2, 4, 8, 16, 32 or another number of high speed interfaces  108 . Each high speed interface  108  may implement a physical communication protocol. In one non-limiting example, each high speed interface  108  may implement the media access control (MAC) protocol, and thus may have a unique MAC address associated with it. The physical communication may be implemented in a known communication technology, for example, Gigabit Ethernet, or any other existing or future-developed communication technology. In one non-limiting example, each high speed interface  108  may implement bi-directional high-speed serial ports, such as 10 Giga bits per second (Gbps) serial ports. Two processing devices  102  implementing such high speed interfaces  108  may be directly coupled via one pair or multiple pairs of the high speed interfaces  108 , with each pair comprising one high speed interface  108  on one processing device  102  and another high speed interface  108  on the other processing device  102 . 
     Data communication between different computing resources of the computing system  100  may be implemented using routable packets. The computing resources may comprise device level resources such as a device controller  106 , cluster level resources such as a cluster controller or cluster memory controller, and/or the processing engine level resources such as individual processing engines and/or individual processing engine memory controllers. An exemplary packet  140  according to the present disclosure is shown in  FIG. 5 . The packet  140  may comprise a header  142  and a payload  144 . The header  142  may include a routable destination address for the packet  140 . The router  104  may be a top-most router configured to route packets on each processing device  102 . The router  104  may be a programmable router. That is, the routing information used by the router  104  may be programmed and updated. In one non-limiting embodiment, the router  104  may be implemented using an address resolution table (ART) or Look-up table (LUT) to route any packet it receives on the high speed interfaces  108 , or any of the internal interfaces interfacing the device controller  106  or clusters  110 . For example, depending on the destination address, a packet  140  received from one cluster  110  may be routed to a different cluster  110  on the same processing device  102 , or to a different processing device  102 ; and a packet  140  received from one high speed interface  108  may be routed to a cluster  110  on the processing device or to a different processing device  102 . 
     The device controller  106  may control the operation of the processing device  102  from power on through power down. The device controller  106  may comprise a device controller processor, one or more registers and a device controller memory space. The device controller processor may be any existing or future-developed microcontroller. In one embodiment, for example, an ARM® Cortex M0 microcontroller may be used for its small footprint and low power consumption. In another embodiment, a bigger and more powerful microcontroller may be chosen if needed. The one or more registers may include one to hold a device identifier (DEVID) for the processing device  102  after the processing device  102  is powered up. The DEVID may be used to uniquely identify the processing device  102  in the computing system  100 . In one non-limiting embodiment, the DEVID may be loaded on system start from a non-volatile storage, for example, a non-volatile internal storage on the processing device  102  or a non-volatile external storage. The device controller memory space may include both read-only memory (ROM) and random access memory (RAM). In one non-limiting embodiment, the ROM may store bootloader code that during a system start may be executed to initialize the processing device  102  and load the remainder of the boot code through a bus from outside of the device controller  106 . The instructions for the device controller processor, also referred to as the firmware, may reside in the RAM after they are loaded during the system start. 
     The registers and device controller memory space of the device controller  106  may be read and written to by computing resources of the computing system  100  using packets. That is, they are addressable using packets. As used herein, the term “memory” may refer to RAM, SRAM, DRAM, eDRAM, SDRAM, volatile memory, non-volatile memory, and/or other types of electronic memory. For example, the header of a packet may include a destination address such as DEVID:PADDR, of which the DEVID may identify the processing device  102  and the PADDR may be an address for a register of the device controller  106  or a memory location of the device controller memory space of a processing device  102 . In some embodiments, a packet directed to the device controller  106  may have a packet operation code, which may be referred to as packet opcode or just opcode to indicate what operation needs to be performed for the packet. For example, the packet operation code may indicate reading from or writing to the storage location pointed to by PADDR. It should be noted that the device controller  106  may also send packets in addition to receiving them. The packets sent by the device controller  106  may be self-initiated or in response to a received packet (e.g., a read request). Self-initiated packets may include for example, reporting status information, requesting data, etc. 
     In one embodiment, a plurality of clusters  110  on a processing device  102  may be grouped together.  FIG. 1B  shows a block diagram of another exemplary processing device  102 A according to the present disclosure. The exemplary processing device  102 A is one particular embodiment of the processing device  102 . Therefore, the processing device  102  referred to in the present disclosure may include any embodiments of the processing device  102 , including the exemplary processing device  102 A. As shown on  FIG. 1B , a plurality of clusters  110  may be grouped together to form a super cluster  130  and an exemplary processing device  102 A may comprise a plurality of such super clusters  130 . In one embodiment, a processing device  102  may include 2, 4, 8, 16, 32 or another number of clusters  110 , without further grouping the clusters  110  into super clusters. In another embodiment, a processing device  102  may include 2, 4, 8, 16, 32 or another number of super clusters  130  and each super cluster  130  may comprise a plurality of clusters. 
       FIG. 2A  shows a block diagram of an exemplary computing system  100 A according to the present disclosure. The computing system  100 A may be one exemplary embodiment of the computing system  100  of  FIG. 1A . The computing system  100 A may comprise a plurality of processing devices  102  designated as F 1 , F 2 , F 3 , F 4 , F 5 , F 6 , F 7  and F 8 . As shown in  FIG. 2A , each processing device  102  may be directly coupled to one or more other processing devices  102 . For example, F 4  may be directly coupled to F 1 , F 3  and F 5 ; and F 7  may be directly coupled to F 1 , F 2  and F 8 . Within computing system  100 A, one of the processing devices  102  may function as a host for the whole computing system  100 A. The host may have a unique device ID that every processing devices  102  in the computing system  100 A recognizes as the host. For example, any processing devices  102  may be designated as the host for the computing system  100 A. In one non-limiting example, F 1  may be designated as the host and the device ID for F 1  may be set as the unique device ID for the host. 
     In another embodiment, the host may be a computing device of a different type, such as a computer processor known in the art (for example, an ARM® Cortex or Intel® x86 processor) or any other existing or future-developed processors. In this embodiment, the host may communicate with the rest of the system  100 A through a communication interface, which may represent itself to the rest of the system  100 A as the host by having a device ID for the host. 
     The computing system  100 A may implement any appropriate techniques to set the DEVIDs, including the unique DEVID for the host, to the respective processing devices  102  of the computing system  100 A. In one exemplary embodiment, the DEVIDs may be stored in the ROM of the respective device controller  106  for each processing devices  102  and loaded into a register for the device controller  106  at power up. In another embodiment, the DEVIDs may be loaded from an external storage. In such an embodiment, the assignments of DEVIDs may be performed offline, and may be changed offline from time to time or as appropriate. Thus, the DEVIDs for one or more processing devices  102  may be different each time the computing system  100 A initializes. Moreover, the DEVIDs stored in the registers for each device controller  106  may be changed at runtime. This runtime change may be controlled by the host of the computing system  100 A. For example, after the initialization of the computing system  100 A, which may load the pre-configured DEVIDs from ROM or external storage, the host of the computing system  100 A may reconfigure the computing system  100 A and assign different DEVIDs to the processing devices  102  in the computing system  100 A to overwrite the initial DEVIDs in the registers of the device controllers  106 . 
       FIG. 2B  is a block diagram of a topology of another exemplary system  100 B according to the present disclosure. The computing system  100 B may be another exemplary embodiment of the computing system  100  of  FIG. 1  and may comprise a plurality of processing devices  102  (designated as P 1  through P 16  on  FIG. 2B ), a bus  202  and a processing device P_Host. Each processing device of P 1  through P 16  may be directly coupled to another processing device of P 1  through P 16  by a direct link between them. At least one of the processing devices P 1  through P 16  may be coupled to the bus  202 . As shown in  FIG. 2B , the processing devices P 8 , P 5 , P 10 , P 13 , P 15  and P 16  may be coupled to the bus  202 . The processing device P_Host may be coupled to the bus  202  and may be designated as the host for the computing system  100 B. In the exemplary system  100 B, the host may be a computer processor known in the art (for example, an ARM® Cortex or Intel® x86 processor) or any other existing or future-developed processors. The host may communicate with the rest of the system  100 B through a communication interface coupled to the bus and may represent itself to the rest of the system  100 B as the host by having a device ID for the host. 
       FIG. 3A  shows a block diagram of an exemplary cluster  110  according to the present disclosure. The exemplary cluster  110  may comprise a router  112 , a cluster controller  116 , an auxiliary instruction processor (AIP)  114 , a cluster memory  118  and a plurality of processing engines  120 . The router  112  may be coupled to an upstream router to provide interconnection between the upstream router and the cluster  110 . The upstream router may be, for example, the router  104  of the processing device  102  if the cluster  110  is not part of a super cluster  130 . 
     The exemplary operations to be performed by the router  112  may include receiving a packet destined for a resource within the cluster  110  from outside the cluster  110  and/or transmitting a packet originating within the cluster  110  destined for a resource inside or outside the cluster  110 . A resource within the cluster  110  may be, for example, the cluster memory  118  or any of the processing engines  120  within the cluster  110 . A resource outside the cluster  110  may be, for example, a resource in another cluster  110  of the computer device  102 , the device controller  106  of the processing device  102 , or a resource on another processing device  102 . In some embodiments, the router  112  may also transmit a packet to the router  104  even if the packet may target a resource within itself. In one embodiment, the router  104  may implement a loopback path to send the packet back to the originating cluster  110  if the destination resource is within the cluster  110 . 
     The cluster controller  116  may send packets, for example, as a response to a read request, or as unsolicited data sent by hardware for error or status report. The cluster controller  116  may also receive packets, for example, packets with opcodes to read or write data. In one embodiment, the cluster controller  116  may be any existing or future-developed microcontroller, for example, one of the ARM® Cortex-M microcontroller and may comprise one or more cluster control registers (CCRs) that provide configuration and control of the cluster  110 . In another embodiment, instead of using a microcontroller, the cluster controller  116  may be custom made to implement any functionalities for handling packets and controlling operation of the router  112 . In such an embodiment, the functionalities may be referred to as custom logic and may be implemented, for example, by FPGA or other specialized circuitry. Regardless of whether it is a microcontroller or implemented by custom logic, the cluster controller  116  may implement a fixed-purpose state machine encapsulating packets and memory access to the CCRs. 
     Each cluster memory  118  may be part of the overall addressable memory of the computing system  100 . That is, the addressable memory of the computing system  100  may include the cluster memories  118  of all clusters of all devices  102  of the computing system  100 . The cluster memory  118  may be a part of the main memory shared by the computing system  100 . In some embodiments, any memory location within the cluster memory  118  may be addressed by any processing engine within the computing system  100  by a physical address. The physical address may be a combination of the DEVID, a cluster identifier (CLSID) and a physical address location (PADDR) within the cluster memory  118 , which may be formed as a string of bits, such as, for example, DEVID:CLSID:PADDR. The DEVID may be associated with the device controller  106  as described above and the CLSID may be a unique identifier to uniquely identify the cluster  110  within the local processing device  102 . It should be noted that in at least some embodiments, each register of the cluster controller  116  may also be assigned a physical address (PADDR). Therefore, the physical address DEVID:CLSID:PADDR may also be used to address a register of the cluster controller  116 , in which PADDR may be an address assigned to the register of the cluster controller  116 . 
     In some other embodiments, any memory location within the cluster memory  118  may be addressed by any processing engine within the computing system  100  by a virtual address. The virtual address may be a combination of a DEVID, a CLSID and a virtual address location (ADDR), which may be formed as a string of bits, such as, for example, DEVID:CLSID:ADDR. The DEVID and CLSID in the virtual address may be the same as in the physical addresses. 
     In one embodiment, the width of ADDR may be specified by system configuration. For example, the width of ADDR may be loaded into a storage location convenient to the cluster memory  118  during system start and/or changed from time to time when the computing system  100  performs a system configuration. To convert the virtual address to a physical address, the value of ADDR may be added to a base physical address value (BASE). The BASE may also be specified by system configuration as the width of ADDR and stored in a location convenient to a memory controller of the cluster memory  118 . In one example, the width of ADDR may be stored in a first register and the BASE may be stored in a second register in the memory controller. Thus, the virtual address DEVID:CLSID:ADDR may be converted to a physical address as DEVID:CLSID:ADDR+BASE. Note that the result of ADDR+BASE has the same width as the longer of the two. 
     The address in the computing system  100  may be 8 bits, 16 bits, 32 bits, 64 bits, or any other number of bits wide. In one non-limiting example, the address may be 32 bits wide. The DEVID may be 10, 15, 20, 25 or any other number of bits wide. The width of the DEVID may be chosen based on the size of the computing system  100 , for example, how many processing devices  102  the computing system  100  has or may be designed to have. In one non-limiting example, the DEVID may be 20 bits wide and the computing system  100  using this width of DEVID may contain up to 2 20  processing devices  102 . The width of the CLSID may be chosen based on how many clusters  110  the processing device  102  may be designed to have. For example, the CLSID may be 3, 4, 5, 6, 7, 8 bits or any other number of bits wide. In one non-limiting example, the CLSID may be 5 bits wide and the processing device  102  using this width of CLSID may contain up to 2 5  clusters. The width of the PADDR for the cluster level may be 20, 30 or any other number of bits. In one non-limiting example, the PADDR for the cluster level may be 27 bits and the cluster  110  using this width of PADDR may contain up to 2 27  memory locations and/or addressable registers. Therefore, in some embodiments, if the DEVID may be 20 bits wide, CLSID may be 5 bits and PADDR may have a width of 27 bits, a physical address DEVID:CLSID:PADDR or DEVID:CLSID:ADDR+BASE may be 52 bits. 
     For performing the virtual to physical memory conversion, the first register (ADDR register) may have 4, 5, 6, 7 bits or any other number of bits. In one non-limiting example, the first register may be 5 bits wide. If the value of the 5 bits register is four (4), the width of ADDR may be 4 bits; and if the value of 5 bits register is eight (8), the width of ADDR will be 8 bits. Regardless of ADDR being 4 bits or 8 bits wide, if the PADDR for the cluster level may be 27 bits then BASE may be 27 bits, and the result of ADDR+BASE may still be a 27 bits physical address within the cluster memory  118 . 
       FIG. 3A  shows that a cluster  110  may comprise one cluster memory  118 . In another embodiment, a cluster  110  may comprise a plurality of cluster memories  118  that each may comprise a memory controller and a plurality of memory banks, respectively. Moreover, in yet another embodiment, a cluster  110  may comprise a plurality of cluster memories  118  and these cluster memories  118  may be connected together via a router that may be downstream of the router  112 . 
     The AIP  114  may be a special processing engine shared by all processing engines  120  of one cluster  110 . In one example, the AIP  114  may be implemented as a coprocessor to the processing engines  120 . For example, the AIP  114  may implement less commonly used instructions such as some floating point arithmetic, including but not limited to, one or more of addition, subtraction, multiplication, division and square root, etc. As shown in  FIG. 3A , the AIP  114  may be coupled to the router  112  directly and may be configured to send and receive packets via the router  112 . As a coprocessor to the processing engines  120  within the same cluster  110 , although not shown in  FIG. 3A , the AIP  114  may also be coupled to each processing engines  120  within the same cluster  110  directly. In one embodiment, a bus shared by all the processing engines  120  within the same cluster  110  may be used for communication between the AIP  114  and all the processing engines  120  within the same cluster  110 . In another embodiment, a multiplexer may be used to control communication between the AIP  114  and all the processing engines  120  within the same cluster  110 . In yet another embodiment, a multiplexer may be used to control access to the bus shared by all the processing engines  120  within the same cluster  110  for communication with the AIP  114 . 
     The grouping of the processing engines  120  on a computing device  102  may have a hierarchy with multiple levels. For example, multiple clusters  110  may be grouped together to form a super cluster.  FIG. 3B  is a block diagram of an exemplary super cluster  130  according to the present disclosure. As shown on  FIG. 3B , a plurality of clusters  110 A through  110 H may be grouped into an exemplary super cluster  130 . Although 8 clusters are shown in the exemplary super cluster  130  on  FIG. 3B , the exemplary super cluster  130  may comprise 2, 4, 8, 16, 32 or another number of clusters  110 . The exemplary super cluster  130  may comprise a router  134  and a super cluster controller  132 , in addition to the plurality of clusters  110 . The router  134  may be configured to route packets among the clusters  110  and the super cluster controller  132  within the super cluster  130 , and to and from resources outside the super cluster  130  via a link to an upstream router. In an embodiment in which the super cluster  130  may be used in a processing device  102 A, the upstream router for the router  134  may be the top level router  104  of the processing device  102 A and the router  134  may be an upstream router for the router  112  within the cluster  110 . In one embodiment, the super cluster controller  132  may implement CCRs, may be configured to receive and send packets, and may implement a fixed-purpose state machine encapsulating packets and memory access to the CCRs, and the super cluster controller  132  may be implemented similar to the cluster controller  116 . In another embodiment, the super cluster  130  may be implemented with just the router  134  and may not have a super cluster controller  132 . 
     An exemplary cluster  110  according to the present disclosure may include 2, 4, 8, 16, 32 or another number of processing engines  120 .  FIG. 3A  shows an example of a plurality of processing engines  120  been grouped into a cluster  110  and  FIG. 3B  shows an example of a plurality of clusters  110  been grouped into a super cluster  130 . Grouping of processing engines is not limited to clusters or super clusters. In one embodiment, more than two levels of grouping may be implemented and each level may have its own router and controller. 
       FIG. 4  shows a block diagram of an exemplary processing engine  120  according to the present disclosure. As shown in  FIG. 4 , the processing engine  120  may comprise an engine core  122 , an engine memory  124  and a packet interface  126 . The processing engine  120  may be coupled to an AIP  114 . As described herein, the AIP  114  may be shared by all processing engines  120  within a cluster  110 . The processing core  122  may be a central processing unit (CPU) with an instruction set and may implement some or all features of modern CPUs, such as, for example, a multi-stage instruction pipeline, one or more arithmetic logic units (ALUs), a floating point unit (FPU) or any other existing or future-developed CPU technology. The instruction set may comprise one instruction set for the ALU to perform arithmetic and logic operations, and another instruction set for the FPU to perform floating point operations. In one embodiment, the FPU may be a completely separate execution unit containing a multi-stage, single-precision floating point pipeline. When an FPU instruction reaches the instruction pipeline of the processing engine  120 , the instruction and its source operand(s) may be dispatched to the FPU. 
     The instructions of the instruction set may implement the arithmetic and logic operations and the floating point operations, such as those in the INTEL® x86 instruction set, using a syntax similar or different from the x86 instructions. In some embodiments, the instruction set may include customized instructions. For example, one or more instructions may be implemented according to the features of the computing system  100 . In one example, one or more instructions may cause the processing engine executing the instructions to generate packets directly with system wide addressing. In another example, one or more instructions may have a memory address located anywhere in the computing system  100  as an operand. In such an example, a memory controller of the processing engine executing the instruction may generate packets according to the memory address being accessed. 
     The engine memory  124  may comprise a program memory, a register file comprising one or more general purpose registers, one or more special registers and one or more events registers. The program memory may be a physical memory for storing instructions to be executed by the processing core  122  and data to be operated upon by the instructions. In some embodiments, portions of the program memory may be disabled and powered down for energy savings. For example, a top half or a bottom half of the program memory may be disabled to save energy when executing a program small enough that less than half of the storage may be needed. The size of the program memory may be 1 thousand (1K), 2K, 3K, 4K, or any other number of storage units. The register file may comprise 128, 256, 512, 1024, or any other number of storage units. In one non-limiting example, the storage unit may be 32-bit wide, which may be referred to as a longword, and the program memory may comprise 2K 32-bit longwords and the register file may comprise 256 32-bit registers. 
     The register file may comprise one or more general purpose registers for the processing core  122 . The general purpose registers may serve functions that are similar or identical to the general purpose registers of an x86 architecture CPU. 
     The special registers may be used for configuration, control and/or status. Exemplary special registers may include one or more of the following registers: a program counter, which may be used to point to the program memory address where the next instruction to be executed by the processing core  122  is stored; and a device identifier (DEVID) register storing the DEVID of the processing device  102 . 
     In one exemplary embodiment, the register file may be implemented in two banks—one bank for odd addresses and one bank for even addresses—to permit fast access during operand fetching and storing. The even and odd banks may be selected based on the least-significant bit of the register address for if the computing system  100  is implemented in little endian or on the most-significant bit of the register address if the computing system  100  is implemented in big-endian. 
     The engine memory  124  may be part of the addressable memory space of the computing system  100 . That is, any storage location of the program memory, any general purpose register of the register file, any special register of the plurality of special registers and any event register of the plurality of events registers may be assigned a memory address PADDR. Each processing engine  120  on a processing device  102  may be assigned an engine identifier (ENGINE ID), therefore, to access the engine memory  124 , any addressable location of the engine memory  124  may be addressed by DEVID:CLSID:ENGINE ID:PADDR. In one embodiment, a packet addressed to an engine level memory location may include an address formed as DEVID:CLSID:ENGINE ID:EVENTS:PADDR, in which EVENTS may be one or more bits to set event flags in the destination processing engine  120 . It should be noted that when the address is formed as such, the events need not form part of the physical address, which is still DEVID:CLSID:ENGINE ID:PADDR. In this form, the events bits may identify one or more event registers to be set but these events bits may be separate from the physical address being accessed. 
     The packet interface  126  may comprise a communication port for communicating packets of data. The communication port may be coupled to the router  112  and the cluster memory  118  of the local cluster. For any received packets, the packet interface  126  may directly pass them through to the engine memory  124 . In some embodiments, a processing device  102  may implement two mechanisms to send a data packet to a processing engine  120 . For example, a first mechanism may use a data packet with a read or write packet opcode. This data packet may be delivered to the packet interface  126  and handled by the packet interface  126  according to the packet opcode. The packet interface  126  may comprise a buffer to hold a plurality of storage units, for example, 1K, 2K, 4K, or 8K or any other number. In a second mechanism, the engine memory  124  may further comprise a register region to provide a write-only, inbound data interface, which may be referred to a mailbox. In one embodiment, the mailbox may comprise two storage units that each can hold one packet at a time. The processing engine  120  may have a event flag, which may be set when a packet has arrived at the mailbox to alert the processing engine  120  to retrieve and process the arrived packet. When this packet is being processed, another packet may be received in the other storage unit but any subsequent packets may be buffered at the sender, for example, the router  112  or the cluster memory  118 , or any intermediate buffers. 
     In various embodiments, data request and delivery between different computing resources of the computing system  100  may be implemented by packets.  FIG. 5  illustrates a block diagram of an exemplary packet  140  according to the present disclosure. As shown in  FIG. 5 , the packet  140  may comprise a header  142  and an optional payload  144 . The header  142  may comprise a single address field, a packet opcode (POP) field and a size field. The single address field may indicate the address of the destination computing resource of the packet, which may be, for example, an address at a device controller level such as DEVID:PADDR, an address at a cluster level such as a physical address DEVID:CLSID:PADDR or a virtual address DEVID:CLSID:ADDR, or an address at a processing engine level such as DEVID:CLSID:ENGINE ID:PADDR or DEVID:CLSID:ENGINE ID:EVENTS:PADDR. The POP field may include a code to indicate an operation to be performed by the destination computing resource. Exemplary operations in the POP field may include read (to read data from the destination) and write (to write data (e.g., in the payload  144 ) to the destination). 
     In some embodiments, the exemplary operations in the POP field may further include bulk data transfer. For example, certain computing resources may implement a direct memory access (DMA) feature. Exemplary computing resources that implement DMA may include a cluster memory controller of each cluster memory  118 , a memory controller of each engine memory  124 , and a memory controller of each device controller  106 . Any two computing resources that implemented the DMA may perform bulk data transfer between them using packets with a packet opcode for bulk data transfer. 
     In addition to bulk data transfer, in some embodiments, the exemplary operations in the POP field may further include transmission of unsolicited data. For example, any computing resource may generate a status report or incur an error during operation, the status or error may be reported to a destination using a packet with a packet opcode indicating that the payload  144  contains the source computing resource and the status or error data. 
     The POP field may be 2, 3, 4, 5 or any other number of bits wide. In some embodiments, the width of the POP field may be selected depending on the number of operations defined for packets in the computing system  100 . Also, in some embodiments, a packet opcode value can have different meaning based on the type of the destination computer resources that receives it. By way of example and not limitation, for a three-bit POP field, a value 001 may be defined as a read operation for a processing engine  120  but a write operation for a cluster memory  118 . 
     In some embodiments, the header  142  may further comprise an addressing mode field and an addressing level field. The addressing mode field may contain a value to indicate whether the single address field contains a physical address or a virtual address that may need to be converted to a physical address at a destination. The addressing level field may contain a value to indicate whether the destination is at a device, cluster memory or processing engine level. 
     The payload  144  of the packet  140  is optional. If a particular packet  140  does not include a payload  144 , the size field of the header  142  may have a value of zero. In some embodiments, the payload  144  of the packet  140  may contain a return address. For example, if a packet is a read request, the return address for any data to be read may be contained in the payload  144 . 
       FIG. 6  is a flow diagram showing an exemplary process  200  of addressing a computing resource using a packet according to the present disclosure. An exemplary embodiment of the computing system  100  may have one or more processing devices configured to execute some or all of the operations of exemplary process  600  in response to instructions stored electronically on an electronic storage medium. The one or more processing devices may include one or more devices configured through hardware, firmware, and/or software to be specifically designed for execution of one or more of the operations of exemplary process  600 . 
     The exemplary process  600  may start with block  602 , at which a packet may be generated at a source computing resource of the exemplary embodiment of the computing system  100 . The source computing resource may be, for example, a device controller  106 , a cluster controller  118 , a super cluster controller  132  if super cluster is implemented, an AIP  114 , a memory controller for a cluster memory  118 , or a processing engine  120 . The generated packet may be an exemplary embodiment of the packet  140  according to the present disclosure. From block  602 , the exemplary process  600  may continue to the block  604 , where the packet may be transmitted to an appropriate router based on the source computing resource that generated the packet. For example, if the source computing resource is a device controller  106 , the generated packet may be transmitted to a top level router  104  of the local processing device  102 ; if the source computing resource is a cluster controller  116 , the generated packet may be transmitted to a router  112  of the local cluster  110 ; if the source computing resource is a memory controller of the cluster memory  118 , the generated packet may be transmitted to a router  112  of the local cluster  110 , or a router downstream of the router  112  if there are multiple cluster memories  118  coupled together by the router downstream of the router  112 ; and if the source computing resource is a processing engine  120 , the generated packet may be transmitted to a router of the local cluster  110  if the destination is outside the local cluster and to a memory controller of the cluster memory  118  of the local cluster  110  if the destination is within the local cluster. 
     At block  606 , a route for the generated packet may be determined at the router. As described herein, the generated packet may comprise a header that includes a single destination address. The single destination address may be any addressable location of a uniform memory space of the computing system  100 . The uniform memory space may be an addressable space that covers all memories and registers for each device controller, cluster controller, super cluster controller if super cluster is implemented, cluster memory and processing engine of the computing system  100 . In some embodiments, the addressable location may be part of a destination computing resource of the computing system  100 . The destination computing resource may be, for example, another device controller  106 , another cluster controller  118 , a memory controller for another cluster memory  118 , or another processing engine  120 , which is different from the source computing resource. The router that received the generated packet may determine the route for the generated packet based on the single destination address. At block  608 , the generated packet may be routed to its destination computing resource. 
       FIG. 7  illustrates an exemplary processing device  102 B according to the present disclosure. The exemplary processing device  102 B may be one particular embodiment of the processing device  102 . Therefore, the processing device  102  referred to in the present disclosure may include any embodiments of the processing device  102 , including the exemplary processing devices  102 A and  120 B. The exemplary processing device  102 B may be used in any embodiments of the computing system  100 . As shown in  FIG. 7 , the exemplary processing device  102 B may comprise the device controller  106 , router  104 , one or more super clusters  130 , one or more clusters  110 , and a plurality of processing engines  120  as described herein. The super clusters  130  may be optional, and thus are shown in dashed lines. 
     Certain components of the exemplary processing device  102 B may comprise buffers. For example, the router  104  may comprise buffers  204 A- 204 C, the router  134  may comprise buffers  209 A- 209 C, the router  112  may comprise buffers  215 A- 215 H. Each of the processing engines  120 A- 120 H may have an associated buffer  225 A- 225 H respectively.  FIG. 8  shows an alternative embodiment of the processing engines  120 A- 120 H such that the buffers  225 A- 225 H may be incorporated into its associated processing engines  120 A- 102 H. Combinations of the implementation of cluster  110  depicted in  FIGS. 7 and 8  are considered within the scope of this disclosure. Also as shown in  FIGS. 7 and 8 , each processing engines  120 A- 120 H may comprise a register  229 A- 229 H respectively. In one embodiment, each of the registers  229 A- 229 H may be a register. In another embodiment, each of the registers  229 A- 229 H may be a register bit. Although one register  229  is shown in each processing engine, the register  229  may represent a plurality of registers for event signaling purposes. In some implementations, all or some of the same components may be implemented in multiple chips, and/or within a network of components that is not confined to a single chip. Connections between components as depicted in  FIG. 7  and  FIG. 8  may include examples of data and/or control connections within the exemplary processing device  102 B, but are not intended to be limiting in any way. Further, as shown in  FIGS. 7 and 8 , each processing engines  120 A- 120 H may comprise a buffer  225 A- 225 H respectively, in one embodiment, each processing engines  120 A- 120 H may comprise two or more buffers. 
     As used herein, buffers may be configured to accommodate communication between different components within a computing system. Alternatively, and/or simultaneously, buffers may include electronic storage, including but not limited to non-transient electronic storage. Examples of buffers may include, but are not limited to, queues, first-in-first-out buffers, stacks, first-in-last-out buffers, last-in-first-out buffers, registers, scratch memories, random-access memories, caches, on-chip communication fabric, switches, switch fabric, interconnect infrastructure, repeaters, and/or other structures suitable to accommodate communication within a multi-core computing system and/or support storage of information. An element within a computing system that serves a purpose as the point of origin for a transfer of information may be referred to as a source. 
     In some implementations, buffers may be configured to store information temporarily, in particular while the information is being transferred from a point of origin, via one or more buffers, to one or more destinations. Structures in the path from a source to a buffer, including the source, may be referred to as being upstream of the buffer. Structures in the path from a buffer to a destination, including the destination, may be referred to as being downstream of the buffer. The terms upstream and downstream may be used as directions and/or as adjectives. In some implementations, individual buffers, such as but not limited to buffers  225 , may be configured to accommodate communication for a particular processing engine, between two particular processing engines, and/or among a set of processing engines. Individual ones of the one or more particular buffers may have a particular status, event, and/or activity associated therewith, jointly referred to as an event. 
     By way of non-limiting example, events may include a buffer becoming completely full, a buffer becoming completely empty, a buffer exceeding a threshold level of fullness or emptiness (this may be referred to as a watermark), a buffer experiencing an error condition, a buffer operating in a particular mode of operation, at least some of the functionality of a buffer being turned on or off, a particular type of information being stored in a buffer, particular information being stored in a buffer, a particular level of activity, or lack thereof, upstream and/or downstream of a buffer, and/or other events. In some implementations, a lack of activity may be conditioned on a duration of idleness meeting or exceeding a particular duration, e.g. a programmable duration. 
     In some implementations, idleness may be indicated by a buffer being empty, a lack of requests for information from downstream structures of a buffer, a lack of information coming in from upstream structures of a buffer, and/or other ways to indicate idleness of a particular buffer, as well as combinations of multiple ways that indicate a lack of activity. For example, a buffer associated with a processing engine may indicate a lack of activity regardless of whether the buffer is empty because the associated processing engine may execute program code that require some time to finish. In one embodiment, the status of a particular buffer may be set to idle responsive to both the following conditions being met: the buffer is completely empty and there has been a lack of requests for information from downstream structures for at least a predetermined duration. In another embodiment, the status of a particular buffer may be set to idle responsive to the following condition being met: no data been added to or removed from the queue for at least a predetermined duration. Other implementations of idleness are considered within the scope of this disclosure. 
     The particular state of a particular processing engine may further include instructions that effectuate generation of signals (e.g. setting a particular register) and/or information (e.g. generating a packet of information) that indicate a particular status, event, and/or activity of one or more particular buffers. For example, a given processing engine may execute a task according to a given state. The given state may include instructions to monitor the level of activity of two given buffers (the two buffers being used to accommodate communication by the given processing engine within a multi-core computing system). For a particular implementation of idleness, a counter may count how many clock cycles both given buffers are considered idle (and/or considered to have a status corresponding to idleness as used in a particular implementation). If either of the two given buffers becomes active (e.g. information is written into the buffer), the counter may reset to zero. Once the counter reaches a predetermined number of clock cycles, both given buffers are deemed to lack activity. The predetermined number of clock cycles may correspond to a particular duration of time. Responsive to the counter reaching the predetermined number, a particular event may be generated (e.g., a particular register may be set to a value that indicates that the related buffers lack activity, at least for the particular duration of time). The particular event may be used elsewhere within the multi-core computing system, e.g. to initiate the process of assigning a new task to the given processing engine. In one embodiment, instead of setting a particular register, a single register bit may be set to indicate occurrence of an event. 
     In some implementations, a set of conditions may be combined in a logical combination to generate a signal (e.g. setting a particular register or a register bit) and/or information (e.g. generating a packet of information) that indicate a particular status. One or more of the conditions of such a set of conditions may be unrelated to idleness. For example, a condition may be that a particular point in a program and/or a particular task in an application has been reached, initiated, and/or completed. In addition to a logical combination, in some implementations, a set of conditions may be a temporal or sequential combination. For example, a first particular event may need to occur prior to the occurrence of a second particular event, and/or both particular events may need to occur subsequent to the occurrence of a third particular event, and so forth. Combinations of logical and sequential events and/or conditions are envisioned within the scope of this disclosure. 
     In embodiments according to the present disclosure, multiple processing engines may be configured to run related processes and/or unrelated processes. For example, a first processing engine may perform a mathematical function on a first set of data, while a second processing engine may perform a process such as monitoring a stream of data items for a particular value. In some implementations, the processes of both processing engines in this example may be unrelated and/or independent. Alternatively, these processes may be related in one or more ways. For example, the mathematical function may only be performed after the particular value has been found in the process running on the second processing engine. Alternatively, the mathematical function may cease to be performed after the particular value has been found. Alternatively, and/or simultaneously, the mathematical function and the process running on the second processing engine may be started and/or stopped together, for example under control of a process running on a third processing engine. For example, the mathematical function running on the first processing engine, the process running on the second processing engine, and/or other processes may be part of an interconnected set of tasks that form an application. 
     Processes to be executed by one or more processing engines may be nested hierarchically and/or sequentially. For example, a first processing engine may perform a first mathematical function on a first set of data, while a second processing engine may perform a different function on a second set of data that includes—as at least one of its input—one or more results of the first mathematical function (e.g. in some implementations, a set or stream of values may be the result of the first mathematical function). In the latter example, the processes of both processing engines are related and/or dependent, e.g. hierarchically and/or sequentially. 
     By way of illustration, assume for a particular scenario or case that the computing system  100  may assign a sequence of tasks (for example, an application) to the processing engines  120 A and  120 B. Processing engines  120 A and  120 B may need to be synchronized at some point in order for the next task in the sequence of tasks to continue on a processing engine, which could be any of the processing engines  120 A- 120 B, one of the processing engines  120 C- 120 H, a processing engine  120  in a different cluster  110  or a processing engine  120  in a different processing device  102 . In an embodiment, data (program code and/or pieces of information upon which the program code operates) needed to execute the sequence of tasks on either processing engines  120 A or  120 B may come from outside of the cluster  110 . For example, the tasks may be assigned by a host of the computing system  100  and the exemplary processing device  102 B may be part of the computing system  100 . The host may load the tasks, assign the tasks to the processing engines  120 A and  120 B of the processing devices  102 B, and send the data for the assigned tasks to the processing engines  120 A and  120 B respectively. The data to be transmitted may be buffered at the router  104  (e.g., using one or more buffers  240 A- 240 C), at the router  134  (e.g., using one or more buffers  209 A- 209 C) (if super clusters are implemented), at the router  112  (e.g., using one or more buffers  215 A- 215 H) and/or at the destination processing engine  120 A and/or  120 B. 
     During execution of the first task, for example, by the processing engine  120 A, the buffer  225 A may be in frequent use, e.g. to transfer data to the processing engine  120 A, to transfer results, output, and/or other data from the processing engine  120 A to other parts of the computing system  100 , and/or to accommodate other types of communication during the first task. Once processing engine  120 A has completed the task, the activity level of buffer  225 A may drop, e.g. to a level that indicates idleness. In one embodiment, individual register may be used to indicate idleness for individual buffers. A particular register of the processing engine  120 A, for example, the register  229 A may be configured to reflect this idleness. Implementations using other mechanisms to reflect this idleness, e.g. though a signal, interrupt, exception, packet, and/or other mechanism, are also within the scope of the present disclosure. It should be noted that although the register  229 A is shown to be within the processing  120 A, it may also be within the buffer  225 A, and/or elsewhere within the computing system  100 , depending on the particular implementation of the computing system  100 . 
     The task or process running on the processing engine  120 B may, at some point, be notified and/or discover that the buffer  225 A is considered idle (and/or considered to have a status corresponding to idleness as used in a particular implementation). In other words, in some implementations, this information may be pushed and/or pulled. Such notification may for example be implemented as an event flag, which may be set in another part of the processing engine  120 , such as but not limited to another register or register bit, and may be pushed and/or pulled from the processing engine  120  (by the device controller  106  and/or by the host of the computing system  100 ). Other implementations that allow an event related to the processing engine  220 A to become apparent, known, or noticed by the processing engine  220 B are also within the scope of this disclosure. Once the processing engine  120 B has gained the knowledge that the processing engine  120 A appears to be idle (and/or considered to have a status corresponding to idleness as used in a particular implementation), the processing engine  120 B may take appropriate actions, such as but not limited to, resume a task that the processing engine  120 B may have stopped, send data to the processing engine  120 A, coordinate with the processing engine  120 A to work on the next task in the sequence of tasks. 
     Although for ease of explanation the above example describes assigning two tasks to processing engines  120 A and  120 B, it is to be understood that embodiments according to the present disclosure may assign hundreds, thousands, hundreds of thousands (or any number for that matter) of tasks to hundreds, thousands, hundreds of thousands (or any number for that matter) of processing engines at processing engine level, cluster level, super cluster level (if super clusters are implemented) and/or processing engine level. 
     In a variation on the preceding scenario or case, additional buffers may be included in a path from the router  104  to the processing engines  120 , in addition to the buffers  204 ,  209 ,  215  and  225 . For example, buffers  265 A,  265 B, and  265 C may be positioned between the router  134  and clusters  110 A,  110 B, and  110 C, respectively; and buffers  235 A,  235 B, and  235 C may be positioned between the router  104  and super clusters  130 A,  130 B, and  130 C, respectively. The buffers shown at the various levels within the hierarchy of system  100  are merely illustrative, and not intended to be limiting in any way. For example, in one embodiment that has no super cluster  130 , there may be additional buffers between the buffer  204  of the router  104  and buffer  215  of the router  112 . 
     In one embodiment, a particular register (e.g. register  229 A) may be configured to reflect the combined idleness of buffer  225 A and a corresponding buffer  215  (e.g., buffer  215 A). The particular register may be configured to only indicate simultaneous idleness of both buffers when both buffers  225 A and  215 A are considered idle at the same time (and/or considered to both have a status corresponding to idleness as used in a particular implementation), and therefore the implied idleness of processing engine  120 A. In such implementations, the processing engine  120 B that has been assigned tasks in the same sequence of tasks as the processing engine  120 A, the device controller  106  and/or the host of the computing system  100  may be configured to monitor all the pertinent register bits for the particular logical combination that indicates all pertinent buffers are idle, and therefore determine that the processing engine  120 A is idle. Once processing engine  120 B notices that processing engine  120 A appears to be idle, processing engine  220 B may take appropriate actions, such as but not limited to, resume a task that the processing engine  120 B may have stopped, send data to the processing engine  120 A, coordinate with the processing engine  120 A to work on the next task in the sequence of tasks, etc. 
     Referring to  FIG. 7 , synchronization among processing engines  120  may be based on, among other features, an ability of individual ones of processing engines  120  to determine whether one or more individual buffers (e.g. buffers  225 ) and/or other components of processing device  102 B may be idle (and/or considered to have a status corresponding to idleness as used in a particular implementation). Individual ones of processing engines  120  may be configured to execute tasks according to a particular current state. The current state may include instructions to be executed. Synchronization between processing engines  120  need not be limited to a single cluster or super cluster, but may extend anywhere within the processing device  102 B and/or between multiple processing devices  102 B. For example, if any of the scenarios described herein where a second processing engine  120  is configured to take appropriate actions upon detection (or being notified) of the processing engine  120 A being idle, the second processing engine  120  may be part of a different cluster  110 , super cluster  130 , or processing device  102 B than processing engine  120 A. 
     Synchronization between processing engines  120  may be based on, among other features, an ability of processing engines  120  to reversibly suspend their own execution, which may be referred to as “going to sleep.” Synchronization between processing engines  120  need not be limited to a single cluster or super cluster, but may extend anywhere within a processing device  102  and/or between multiple processing devices  102  in a computing system  100 . 
     In some implementations, a particular processing engine  120  may be configured to execute one or more instructions (from a set of instructions) that reversibly suspend execution of instructions by that particular processing engine  120 . Other components within a computing system  100 , including but not limited to components at different levels within a hierarchy of a processing device  102 , may be configured to cause such a suspension to be reversed, which may be referred to as “waking up” a (suspended) processing engine. 
     Processing engines  120  may be configured to operate in one or more modes of power consumption, including a low-power mode of consumption (e.g. when the processing engine has gone to sleep) and one or more regular power modes of consumption when execution is not suspended. In some implementations, the low power mode of consumption reduces power usage by a factor of at least ten compared to power usage when execution is not suspended. In some implementations, waking up a processing engine may be implemented as exiting the low-power mode of power consumption. 
     In one embodiment, individual processing engines  120  may generate and send signals to indicate one or more occurrences of one or more events within the individual processing engine  120 . As used herein, signals indicative of events may be referred to as event signals and the term “event” may also mean the signal representing an occurrence of the event. An event may interchangeably refer to any event, status, activity (or inactivity) of a processing element of the computing system according to the present disclosure. For example, an event may be related to and/or associated with an access of a memory or a buffer within individual processing engine  120 , including but not limited to a read access of a memory, a write access of a memory, a busy signal for a memory arbiter or buffer arbiter, a FIFO-full indication of a first-in-first-out (FIFO) buffer, etc. Alternatively, and/or simultaneously, an event may be related to and/or associated with a delay of processing within an individual processing engine  120 . For example, an event may indicate congestion in data transfer, a status of non-responsiveness, a status that indicates waiting for instructions, data and/or other information, and/or other types of processing delays and/or bottlenecks. Alternatively, and/or simultaneously, an event may be related to and/or associated with a (completion of an) execution of an instruction and/or task within individual processing engine  120 . 
     In some embodiments, whether an event has occurred in a computing resource may be based on an amount of time during which certain activity or activities have or have not occurred. In one embodiment, one or more timing registers may be implemented. For example, such timing registers may be implemented by a processing engine  120 , a cluster  110 , a super cluster  130 , and/or a processing device  102 . One timing register may be used, for example, to record the time since the last activity in one of the buffers, and another timing register may be used to record time since the last activity in another of the buffers. As used herein, a timing register may also be referred to as a timer. The timing register may be implemented at one or more of processing engine level, cluster level, super cluster level and processing device level. In one embodiment, one or more timing registers may be implemented on each of the levels of the hierarchy on a processing device  102 . In another embodiment, one or more timing registers may be implemented on a processing device  102  at the processing device level but may be programmed or configured to be used for each processing engines  120 , clusters  110  and/or super clusters  130  individually. 
     Event signals generated at the processing engine level may be propagated from the processing engine  120  to cluster level, super cluster level, processing device level and/or a host of the computing system  100 . Event signal propagation may be implemented, in a non-limiting example, by multiplexers at cluster level, at super cluster level and/or processing device level. For example, a host of a computing system  100  may receive event signals from all processing devices  102  within the computing system  100 ; a processing device  102  may receive event signals propagated from all super clusters  130  (if super clusters are implemented), cluster  110 , and/or processing engines  120  within the processing device  102 ; a super cluster  130  may receive event signals propagated from all clusters  110  and/or processing engines  120  within the super cluster  130 ; and a cluster  110  may receive event signals propagated from all processing engines  120 . 
     In one embodiment, in addition to propagating event signals, each of the clusters  110 , super clusters  130  and processing devices  102  may generate event signals by itself. For example, a cluster  110  may generate an event signal based on activity levels of a specified subset or all of processing engines  120  within the cluster  110  to indicate an activity level for the cluster as a whole; a super cluster  130  may generate an event signal based on activity levels of a specified subset or all of processing engines  120  within the super cluster  130  to indicate an activity level for the super cluster  130  as a whole; and a processing device  102  may generate an event signal based on activity levels of a specified subset or all of processing engines  120  within the processing device  102  to indicate an activity level for the processing device as a whole. For example, if of all processing engines  120 A- 120 H, only the processing engines  120 A and  120 B may have been assigned tasks to execute, a cluster level event may be generated based on activity levels on the processing engines  120 A and  120 B (e.g., activity levels of buffers  225 A and  225 B) instead of all processing engines  120 A- 120 H in the cluster  110 . 
     The event signals may be stored in event registers at the cluster  110 , super cluster  130  and/or processing devices  102  level, and may be collected by a host in the computer system  100 . It should be noted that, in one embodiment, there may be separate event signals for computation activity (e.g., based on event registers  229  of the processing engines  120 ), network activity (e.g., based on event registers (not shown) of the routers  104 ,  112 , and  134 ) and/or memory access activity (e.g., based on event registers of the cluster memories  118 ). In one embodiment, the event signals may comprise one or more bits and each event signal may be in a “set” or “non-set” value. A “set” value may indicate the event represented by the event signal has occurred and a “non-set” value may indicate that the event represented by the event signal has not occurred. Therefore, the event signal may also be used to represent states of a respective processing element, such as but not limited to, whether a buffer is empty, and whether a buffer has been idle for a certain amount of time. 
     As described herein, event signals may be generated based on timing. For example, one or more processing engines  120 A- 120 H may be assigned some tasks, and the cluster  110  may generate an event signal indicating that the cluster  110  is idle only when all buffers  225 A- 225 H have been idle for a time threshold. The time threshold may be a predetermined amount of time and stored in a programmable register that may be updated from time to time or at appropriate time. The predetermined amount of time may be different or the same for processing engine, cluster, super cluster and processing device levels. For example, a time threshold for determining whether a processing engine  120  is idle may be different from a time threshold for determining whether a cluster, a super cluster, or a processing device is idle. Moreover, the predetermined amount of time may also be different or the same for different components. For example, there may be one time threshold for determining whether a processing engine is idle, another time threshold for determining whether a router  102  is idle, and/or a different time threshold for determining whether a router  112  is idle. 
     In one embodiment, the predetermined amount of time may be set by, for example, a programmer, a system administrator, or a software program that may dynamically adjust parameters for the operation of the computing system  100 . The programmable register for timing may be implemented by one or more registers at the processing device level, the super cluster level, the cluster level and/or processing engine level. In one embodiment, the programmable register for timing may be re-used for different purposes and contain different values for the different purposes. For example, during a certain time period, a programmable register at a processing device  102  may be used for activities within a cluster  110  and during a different timer period, the same programmable register may be used for latency of the buffer  204 . 
     The timers used for counting the time may be implemented in incrementing and/or decrementing manners. In one embodiment, a timer may be started based on one or more event signals. If there is no change to any of the one or more event signals until the counted time reaches the respective time threshold, another event signal may be generated. If, however, anyone of the one or more event signals changes its event signal, the timer may be reset. For example, a timer for the cluster  110  may be started based on the event signals from the processing engines  120 A- 120 H that they are idle and the timer may start when the latest idle event signal is received. If anyone of processing engines  120 A- 120 H changes its state from idle (e.g., from “set” to “non-set”), the timer for the cluster  110  may be reset. If the processing engines  120 A- 120 H maintain their idle states until the timer reaches a predetermined amount of time, an event signal may be generated indicating the cluster  110  is itself idle. In this example, the processing engines  120 A- 120 H may be reduced to a specified subset, for example, only processing engines  120 A and  120 B, if only processing engines  120 A and  120 B may be relevant to determine whether the cluster  110  is idle. Further, in this example, the cluster  110  may be replaced with the super cluster  130  or the processing device  102 , and correspondingly the processing engines  120 A- 120 H may be replaced by a specified subset or all of the processing engines within the super cluster  130  or a specified subset or all of the processing engines within the processing device  102 . 
     Event, status, activity and any other information related to the operating state of a computing system comprising a plurality of processing devices  102  may be generated, counted and/or collected at each of the processing engine, cluster, super cluster, and/or processing device level. The computing system may comprise a host that collects all that information from all computing resources across the computing system.  FIG. 9  illustrates an exemplary host  11  configured to synchronize tasks among processing elements in an exemplary computing system  100 C according to the present disclosure. The exemplary computing system  100 C may be an example of the computing system  100  and may implement all features of the computing system  100  described herein. The host  11  may be an example of a host for the computing system  100  and may implement all features of a host of the computing system  100  described herein. As depicted in  FIG. 9 , the computing system  100 C may comprise a plurality of processing devices  102  in addition to the host  11 . The number of processing devices  102  may be as low as a couple or as high as hundreds of thousands, or even higher limited only by the width of DEVID. The exact number of processing devices  102  is immaterial and thus, the processing devices  102  are shown in phantom. Moreover, in one embodiment, each of the processing devices  102  may be an embodiment of the processing device  102 B as shown in  FIG. 7 . The processing engines  120  on each processing device  102  may implement buffers  225  (as shown in  FIG. 7  or  FIG. 8 ). The host  11  may comprise one or more processors  20 , a physical storage  60 , and an interface  40 . The processing elements that may be assigned tasks may include processing engines  120  and/or processing devices  102 . If a task is assigned to a processing device  102 , the processing device  102  may implement functionality to further assign the task to one of the processing engines  120  of the processing device  102 . In one embodiment, the topology and/or interconnections within the computing system  100 C may be fixed. In another embodiment, the topology and/or interconnections within the computing system  100 C may be programmable. 
     Interface  40  may be configured to provide an interface between the computing system  100 C and a user (e.g., a system administrator) through which the user can provide and/or receive information. This enables data, results, and/or instructions and any other communicable items, collectively referred to as “information,” to be communicated between the user and the computing system  100 C. Examples of interface devices suitable for inclusion in interface  40  include a keypad, buttons, switches, a keyboard, knobs, levers, a display screen, a touch screen, speakers, a microphone, an indicator light, an audible alarm, and a printer. Information may be provided by interface  40  in the form of auditory signals, visual signals, tactile signals, and/or other sensory signals. 
     It is to be understood that other communication techniques, either hard-wired or wireless, are also contemplated herein as interface  40 . For example, in some implementations, interface  40  may be integrated with physical storage  60 . In this example, information is loaded into computing system  100 C from storage (e.g., a smart card, a flash drive, a removable disk, etc.) that enables the user(s) to customize the implementation of computing system  100 C. Other exemplary input devices and techniques adapted for use with computing system  100 C as interface  40  include, but are not limited to, an RS-232 port, RF link, an IR link, modem (telephone, cable, Ethernet, internet or other). In short, any technique for communicating information with computing system  100 C is contemplated as interface  40 . 
     One or more processors  20  (interchangeably referred to herein as processor  20 ) may be configured to execute computer program components. The computer program components may include an assignment component  24 , a loading component  25 , a program component  26 , a performance component  27 , an analysis component  28 , and/or other components. The functionality provided by components  24 - 28  may be attributed for illustrative purposes to one or more particular components of system  100 C. This is not intended to be limiting in any way, and any functionality may be provided by any component or entity described herein. 
     The functionality provided by components  24 - 28  may be used to load and execute one or more computer applications, including but not limited to one or more computer test applications, one or more computer web server applications, or one or more computer database management applications. An application may comprise one or more tasks, e.g. a set of interconnected tasks that jointly form the application. The applications may include test applications used to determine, measure, estimate, debug, and/or monitor the functionality and/or performance of a particular processing engine, cluster, super cluster and/or a multi-core processing system. As used herein, references to a system&#39;s functionality and/or performance are considered to include a system&#39;s design, testing, calibration, configuration, load balancing, and/or operation at any phase during its lifecycle. System  100 C may be configured to divide applications, including but not limited to test application, into sets of interconnected tasks. For example, an application could include software-defined radio (SDR) or some representative portion thereof. For example, a test application could be based on an application such as SDR, for example by scaling down the scope to make testing easier and/or faster. By way of non-limiting example, a SDR application may include one or more of a mixer, a filter, an amplifier, a modulator, a demodulator, a detector, and/or other tasks and/or components that, when interconnected, may form an application. In another example, a software application may comprise a plurality of modules that may be treated as separate tasks, such as but not limited to, dynamic link libraries (DLLs), Java Archive (JAR) packages, and similar libraries on UNIX®, ANDROID® or MAC® operating systems. 
     Loading component  25  may be configured to load, link, and/or program instructions, state, functions, and/or connections into computing system  100 C and/or its components. State may include data, including but not limited to, program code and information upon which the program code may operate for operating the system  100 C (e.g., an operating system) and/or software applications to be executed by the system  100 C. State may also include information regarding interconnections among the host  11  and the processing devices  102 , clusters  110 , super clusters  130  (if super clusters are implemented), and/or set of processing engines  120 , and/or other information needed to execute a particular task (or any other part of a software application). The program code may include instructions that generate signals (and/or effectuate generation of signals) that are indicative of occurrences of particular events, status, and/or activity within processing devices  102  and/or various buffers within the processing devices  102 . In some implementations, the state may be determined by program component  26 . In some implementations, loading component  25  may be configured to load and/or program a set of processing engines  120  and/or buffers (e.g. the same as or similar to processing engines  120  and/or any buffers shown in  FIG. 7 or 8 ), a set of interconnections, and/or additional functionality into system  100 C. For example, additional functionality may include input processing, memory storage, data transfer within one or more processing engines  120 , output processing, and/or other functionality. By virtue of the synchronization mechanisms described in this disclosure, a multi-core processing system such as the computing system  100 C including multiple processing devices  102  and/or processing engines  120  may be more easily configured, partitioned, and/or load-balanced while maintaining functionally correct interoperation between multiple processing engines  120 . In some implementations, loading component  25  may be configured to execute (at least part of) applications, e.g. responsive to functions and/or connections being loaded into system  100 C and/or its components. 
     Assignment component  24  may be configured to assign one or more computing resources within the computing system  100 C to perform one or more tasks. The computing resources that may be assigned tasks may include processing devices  102 , clusters  110 , super clusters  130  (if super clusters are implemented), and/or processing engines  120 . In some implementations, assignment component  24  may be configured to perform assignments in accordance with and/or based on a particular routing. For example, a routing may limit the number of processing devices  102  and/or processing engines  120  that are directly connected to a particular processing engine  120 . In some implementations, by way of non-limiting example, the routing of a network of processing devices  102  may be fixed (i.e. the hardware connections between different processing devices  102  may be fixed), but the assignment of particular tasks to specific computing resources may be refined, improved, and/or optimized in pursuit of higher performance. In some implementations, by way of non-limiting example, the routing of a network of processing engines  102  may not be fixed (i.e. programmable between iterations of performing an assignment and determining the performance of a particular assignment), and the assignment of particular tasks to specific processing devices  102  and/or processing engines  120  may be also be adjusted, e.g. in pursuit of higher performance. 
     Assignment component  24  may be configured to determine and/or perform assignments of tasks repeatedly, e.g. in the pursuit of higher performance. Assignments of tasks may be performed conditional to one or more particular processing engines  120  or processing devices  102  being idle (and/or being considered to have a status corresponding to idleness as used in a particular implementation). As used herein, any association (or correspondence) involving applications, chips, processing engines, tasks, and/or other entities related to the operation of systems described herein, may be a one-to-one association, a one-to-many association, a many-to-one association, and/or a many-to-many association or N-to-M association (note that N and M may be different numbers greater than 1). For example, assignment component  24  may assign one or more processing engines  120  distributed among one or more processing devices  102  to perform the task or tasks of one or more mixers of an SDR application. Assignment of tasks to a combination including one or more processing engines  120  and one or more processing devices  102  may also be envisioned within the scope of this disclosure. 
     Program component  26  may be configured to determine state for processing devices  102 , clusters  110 , super clusters  130  (if super clusters are implemented), and/or processing engines  120 . The particular state for a particular cluster  110 , super cluster  130  (if super clusters are implemented), or processing engine  120  may be in accordance with an assignment and/or routing from another component of system  100 C. In some implementations, program component  26  may be configured to program and/or load instructions and/or state into one or more clusters  110 , super clusters  130  (if super clusters are implemented), and/or processing engines  120 . In some implementations, programming individual processing engines  120 , clusters  110 , super clusters  130  (if super clusters are implemented), and/or processing devices  102  may include setting and/or writing control registers, for example, CCRs for cluster controllers  116  and super cluster controllers  132 , control registers within the device controller  106 , or control registers within the processing engines  120 . 
     By way of illustration, the host  11  may assign a sequence of tasks, e.g. an application formed by interrelated tasks to the processing engine  120 A and processing engine  120 B of one processing device  102  as shown in  FIG. 7 . The processing engine  120 A and  120 B may need to be synchronized at a certain point. For example, assume that a first task in the sequence of tasks is assigned to the processing engine  120 A and a second task in the sequence of tasks is assigned to the processing engine  120 B. While processing engines  120 A and  120 B are busy executing their respective assigned task, they may not be assigned another task. As described herein, the processing engines  120 A and  120 B may comprise buffers  225 A and  225 B respectively for receiving data from and send data to other parts of the device  102  (including other parts of the computing system  110 C via the router  104 ). 
     During execution of the first task by processing engine  120 A, buffer  225 A may be in frequent use, e.g. to receive data sent to processing engine  120 A, to transfer results, output and/or other data from processing engine  120 A to other parts of the processing device  102  (including other part of the computing system  100 C via the router  104 ), and/or to accommodate other types of communication during the first task. During execution of the second task by processing engine  120 B, buffer  225 B may be in frequent use, e.g. to receive data sent to processing engine  120 B, to transfer results, output, and/or other data from processing engine  120 B to other parts of the processing device  102  (including other part of the computing system  100 C via the router  104 ), and/or to accommodate other types of communication during the second task. Once either processing engines  120 A or  120 B has completed its respective task, the activity level of buffer  225 A or  225 B may drop, e.g. to a level that indicates idleness. The register bits  229 A and  229 B may be configured to reflect this idleness respectively. Although  FIGS. 7 and 8  show the registers  229  are located within respective processing engines  120 A- 120 H, in another embodiment, the registers  229  may reside within the buffer  225  and/or elsewhere within system  100 C, depending on the particular implementation of system  100 . Implementations using other mechanisms to reflect idleness, e.g. though a signal, interrupt, exception, packet, and/or other mechanism, are also within the scope of this disclosure. 
     The host  11  may, at some point, be notified and/or discover that buffer  225 A and buffer  225 B are considered idle (and/or considered to have a status corresponding to idleness as used in a particular implementation). In other words, in some implementations, this information may be pushed and/or pulled. Such notification may for example be implemented as an event flag, which may be another part of the processing engine  120  and may be pushed and/or pulled from the processing engine  120  (by the device controller  106  and/or by the host of the computing system  100 ). Other implementations that allow an event related to processing engine  120 A or processing engine  120 B to become apparent, known, or noticed by host  11  are also within the scope of this disclosure. 
     In one embodiment, once host  11  has gained the knowledge that processing engine  120 A and/or processing engine  120 B appear to be idle (and/or considered to have a status corresponding to idleness as used in a particular implementation), host  11  may assign the next one or more tasks in the sequence of tasks to processing engine  120 A or processing engine  120 B for execution, as appropriate in the context of the sequence of tasks, which may be interrelated. Moreover, in one embodiment, the host  11  may also activate a processing engine that may have been idle because it is waiting for certain activity (or activities) to occur first. For example, the processing engine  120 B may need to wait for the processing engine  120 A to finish certain computation task (or portion thereof) before it can start (or resume) processing a task assigned to itself. It should be noted that such a notification may be implemented such that involvement of host  11  may not be necessary. For example, the cluster  110 , the super cluster  130  and/or the processing device  102  may implement a notification to be sent to the related processing engine (in this case processing engine  120 B) when an event of idleness of the processing engine  110 A has occurred 
     In some implementations, host  11  may assign the next task to either processing engine  120 A or  120 B depending on which processing engine has completed their respective previous task first. In some implementations, host  11  may assign the next one or more tasks to both processing engine  120 A and  120 B once both the first task and the second tasks have been completed. In another embodiment, host  11  may designate at least part of its functionality to another processing engine  120  such that this designated processing engine  120  may perform certain functions, such as but not limited to, monitoring the status of the processing engines  120 A and  120 B, assign new tasks from the sequence of tasks to the processing engines  120 A and/or  120 B once either or both of them finish their respective tasks. Different logical combinations and sequential combinations of tasks are envisioned within the scope of this disclosure. 
     Performance component  27  may be configured to determine performance parameters of computing system  100 C, one or more processing devices  102 , one or more clusters  110 , one or more super clusters  130  (if super cluster is implemented), one or more processing engines  120 , and/or other configurations or combinations of processing elements described herein. In some implementations, one or more performance parameters may indicate the performance and/or functionality of an assignment of tasks (and/or a sequence of assignments of tasks), as performed by the computing system  100 C. For example, one or more performance parameters may indicate bottlenecks, speed, delays, and/or other characteristics of performance and/or functionality for computing resources within the system  100 C, such as but not limited to memories, routers, processing engines. In some implementations, performance may be associated with a particular application, e.g. a test application, or a particular strategy used to assign tasks to processors. In some implementations, one or more performance parameters may be based on signals generated within and/or by one or more processing engines  120  or other components of one or more processing devices  102  (including the various buffers shown in  FIG. 7 ) and/or other components of system  100 C. For example, the generated signals may be indicative of occurrences or events within a particular component of system  100 C, as described elsewhere herein. By virtue of the synchronization mechanisms described in this disclosure, the performance of (different configurations and/or different assignments of) multi-core processing systems may be monitored, determined, and/or compared. 
     Analysis component  28  may be configured to analyze performance parameters. In some implementations, analysis component  28  may be configured to compare performance of different configurations of multi-core processing systems, different ways to divide an application into a set of interconnected tasks by a programmer (or a compiler, or an assembler), different assignments by assignment component  24 , and/or other different options used during the configuration, design, and/or operation of a multi-core processing system. 
     In some implementations, analysis component  28  may be configured to indicate a bottleneck and/or other performance issue in terms of memory access, computational load, and/or communication between multiple processing elements/engines. For example, one task may be loaded on a processing engine and executed on it. If the processing engine is kept busy (e.g., no event signal of idleness) for a predetermined amount of time, then the task may be identified as a computation intensive task and a good candidate to be executed in parallel, such as being executed in two or more processing engines. In another example, two processing engines may be assigned to execute some program code respectively (could be one task split between the two processing engines, or each processing engine executing one of two interconnected tasks). If each of the two processing engines spends more than a predetermined percentage of time (e.g., 10%, 20%, 30% or another percentage, which may be programmable) waiting on other processing engine (e.g., for data or an event signal), then the program code may be identified as communication intensive task(s) and a good candidate to be executed on a single processing engine, or moved to be closer (such as but not limited to, two processing engines in one cluster, two processing engines in one super cluster, or two processing engines in one processing device). 
     Referring to  FIG. 9 , one or more processors  20  may be configured to provide information-processing capabilities in computing system  100 C and/or host  11 . As such, processor  20  may include one or more of a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information. Although processor  20  may be shown in  FIG. 9  as a single entity, this is for illustrative purposes only. In one embodiment, processor  20  may include a plurality of processing units. For example, each processor  20  may be a processing device  102  or a processor of a different type as described herein. These processing units may be physically located within the same physical apparatus, or processor  20  may represent processing functionality of a plurality of apparatuses operating in coordination (e.g., “in the cloud”, and/or other virtualized processing solutions). 
     It should be appreciated that although components  24 - 28  are illustrated in  FIG. 9  as being co-located within a single processing unit, in implementations in which processor  20  includes multiple processing units, one or more of components  24 - 28  may be located remotely from the other components. The description of the functionality provided by the different components  24 - 28  described herein is for illustrative purposes, and is not intended to be limiting, as any of components  24 - 28  may provide more or less functionality than is described. For example, one or more of components  24 - 28  may be eliminated, and some or all of its functionality may be provided by other ones of components  24 - 28 . As another example, processor  20  may be configured to execute one or more additional components that may perform some or all of the functionality attributed herein to one of components  24 - 28 . 
     Physical storage  60  of computing system  100 C in  FIG. 9  may comprise electronic storage media that stores information. In some implementations, physical storage  60  may store representations of computer program components, including instructions that implement the computer program components. The electronic storage media of physical storage  60  may include one or both of system storage that is provided integrally (i.e., substantially non-removable) with host  11  and/or removable storage that is removably connectable to host  11  via, for example, a port (e.g., a USB port, a FIREWIRE port, etc.) or a drive (e.g., a disk drive, etc.). Physical storage  60  may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), network-attached storage (NAS), and/or other electronically readable storage media. Physical storage  60  may include virtual storage resources, such as storage resources provided via a cloud and/or a virtual private network. Physical storage  60  may store software algorithms, information determined by processor  20 , information received via client computing platforms  14 , and/or other information that enable host  11  and computing system  100 C to function properly. Physical storage  60  may be one or more separate components within system  100 C, or physical storage  60  may be provided integrally with one or more other components of computing system  100 C (e.g., processor  20 ). 
     Users may interact with system  100 C through client computing platforms  14 . By way of non-limiting example, client computing platforms may include one or more of a desktop computer, a laptop computer, a handheld computer, a NetBook, a Smartphone, a tablet, a mobile computing platform, a gaming console, a television, a device for streaming internet media, and/or other computing platforms. Interaction between the system  100 C and client computing platforms may be supported by one or more networks  13 , including but not limited to the Internet. 
       FIGS. 10 and 11  illustrate exemplary processes  1000  and  1100  of synchronizing processing elements within a processing device and within a multi-core computing system, respectively, according to the present disclosure. Each of the processing elements may be a processing engine  120 , a cluster  110 , a super cluster  130  (if super clusters are implemented) or a processing device  102 . The operations of processes  1000  and  1100  presented below are intended to be illustrative. In some implementations, processes  1000  and  1100  may be accomplished with one or more additional operations not described, and/or without one or more of the operations discussed. Additionally, the order in which the operations of processes  1000  and  1100  are illustrated in  FIGS. 10 and 11  and described below is not intended to be limiting. 
     One example of processing device configured to execute the exemplary process  1000  may be an exemplary embodiment of the processing device  102  (including  102 A or  102 B), in which the various components of the processing device  102 , such as but not limited to, one or more of the processing engines  120 , clusters  110 , and super clusters  130  and the processing device  102  itself, may be configured through hardware, firmware, and/or software to be specifically designed for execution of one or more of the operations of exemplary process  1000 . 
     The exemplary process  1000  may start with block  1002 , at which one or more tasks may be loaded to two or more processing elements of a plurality of processing elements. For example, a processing device  102  may comprise a plurality of processing elements, such as 256 processing engines in one embodiment of the processing device  102 . As described herein, in some embodiments, the processing engines may be grouped into clusters and in one of such embodiments, the clusters may be further grouped into super clusters. Tasks may be assigned, for example, to processing engines, to clusters, to super clusters (if super clusters are implemented), and/or to processing engines. Not all processing elements may be needed to execute some tasks, which may be a software application that can be executed in parallel by executing the tasks of the application in parallel. 
     At block  1004 , the one or more tasks may be executed on the two or more processing elements. For example, if only a subset of the processing engines  120  on a processing device  102  are assigned tasks to execute, then the tasks may be executed on the subset of the processing engines  120 . 
     At block  1006 , buffers associated with the two or more processing elements may be monitored. The monitored buffers may be used to communicate the one or more tasks to the two or more processing elements. As described herein, each processing element of a processing device  102  may have a buffer associated with it for receiving and sending data (including program code and information upon which the program code operate). The processing device  102  may implement an event mechanism (including but not limited to, event registers, timing registers, programmable registers to hold time thresholds) to indicate whether certain activities have occurred. At block  1008 , states of the two or more processing elements may be determined based on the monitored buffer activities. For example, whether a processing engine  120  is in an idle state may be determined based on whether a buffer associated with the processing engine  120  has been idle for a certain amount of time. At block  1010 , a first event flag may be set after no activity is monitored in at least one of the two or more processing elements based on the determined states. Continuing with the preceding example, an event flag may be set after there is no activity is monitored for the processing engine  120  based on the monitored buffer associated with the processing engine  120  having no activity for the certain amount of time. 
     One example of computing system  100  configured to execute the exemplary process  1100  may be the computing system  100 C, in which the host  11  and other components of the computing system  100 C, such as but not limited to, one or more of the processing engines  120 , clusters  110 , super clusters  130  and the processing devices  102 , may be configured through hardware, firmware, and/or software to be specifically designed for execution of one or more of the operations of exemplary process  1100 . 
     The exemplary process  1100  may start with block  1102 , one or more tasks may be assigned to at least a subset of processing elements of a plurality of processing devices in a computing system. For example, the exemplary system  100 C may comprise hundreds of processing devices  102  and each may comprise hundreds of processing engines. A computer application to be executed by the computing system  100 C may have only a couple of tasks to be assigned to two processing engines for parallel processing and the host  11  may assign the couple of tasks to two processing engines, such as the processing engines  120 A and  120 B shown on  FIG. 7 or 8 . At block  1104 , the one or more tasks may be loaded to the assigned processing elements. Continuing with the preceding example, the couple of tasks assigned to the processing engines  120 A and  120 B may be loaded to the processing engines  120 A and  120 B. 
     At block  1106 , the one or more tasks may be executed on the assigned processing elements. For example, the couple of tasks assigned to the processing engines  120 A and  120 B may be executed on the processing engines  120 A and  120 B. At block  1108 , buffers associated with the assigned processing elements may be monitored and the monitored buffers may be used to communicate the one or more tasks to the two or more processing elements. In one embodiment, the host  11  may load the assigned tasks to the processing engines  120 A and  120 B. As described herein, the loading may include sending data to the processing device  102  and via the routers  104 ,  134  (if super clusters are implemented) and  112 . Each processing element, such as the processing engines  120 A and  120 B, of a processing device  102  may have a buffer associated with it for receiving and sending data (including program code and information upon which the program code operate). The processing device  102  may implement an event mechanism (including but not limited to, event registers, timing registers, programmable registers to hold time thresholds) to indicate whether certain activities have occurred. 
     At block  1110 , states of the assigned processing elements may be determined based on the monitored buffer activities. For example, whether a processing engine  120  is in an idle state may be determined based on whether a buffer associated with the processing engine  120  has been idle for a certain amount of time. In one embodiment, the states may be determined at the host  11 , the processing device  102 , the cluster  110 , and/or another processing engine (inside the same cluster  102  or anywhere within the computing system  100 C). 
     At block  1112 , a first event flag may be set after no activity is monitored in at least one of the two or more processing elements based on the determined states. For example, an event flag may be set after there is no activity is monitored for a processing engine  120  based on the monitored buffer  225  associated with the processing engine  120  having no activity for the certain amount of time. 
     It should be noted that some of the operations of processes  1000  and  1100  may be similar because operating a computing system  100  for parallel processing includes identical and/or similar features as operating a processing device  102  for parallel processing. In those operations, the description with respect to one operation in one of the processes  1000  and  1100  may be applicable to the corresponding operation in the other process as well. 
     By virtue of the synchronization mechanisms described in this disclosure, a computing system that supports parallel processing may be more easily configured, partitioned, and/or load-balanced while maintaining functionally correct interoperation between multiple computing resources of the computing system. Non-exclusive examples of computing resources may include processing engines, clusters, super clusters, and/or processing devices. It should be noted that not all computing resources, for example, device controllers, routers, memory controllers, etc., will actually execute program code of computation tasks, but these computing resources may be configured to facilitate the processing engines to coordinate, cooperate and execute program code of computation tasks and they may also be configured to generate event signals to indicate occurrence of events within the respective computing resource. 
     While specific embodiments and applications of the present invention have been illustrated and described, it is to be understood that the invention is not limited to the precise configuration and components disclosed herein. The terms, descriptions and figures used herein are set forth by way of illustration only and are not meant as limitations. Various modifications, changes, and variations which will be apparent to those skilled in the art may be made in the arrangement, operation, and details of the apparatuses, methods and systems of the present invention disclosed herein without departing from the spirit and scope of the invention. By way of non-limiting example, it will be understood that the block diagrams included herein are intended to show a selected subset of the components of each apparatus and system, and each pictured apparatus and system may include other components which are not shown on the drawings. Additionally, those with ordinary skill in the art will recognize that certain steps and functionalities described herein may be omitted or re-ordered without detracting from the scope or performance of the embodiments described herein. 
     The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. The described functionality can be implemented in varying ways for each particular application—such as by using any combination of microprocessors, microcontrollers, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and/or System on a Chip (SoC)—but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention. 
     The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. 
     The methods disclosed herein comprise one or more steps or actions for achieving the described method. The method steps and/or actions may be interchanged with one another without departing from the scope of the present invention. In other words, unless a specific order of steps or actions is required for proper operation of the embodiment, the order and/or use of specific steps and/or actions may be modified without departing from the scope of the present invention.