Patent Publication Number: US-8125984-B2

Title: Method, system, and computer program product for implementing stream processing using a reconfigurable optical switch

Description:
STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT 
     The invention disclosed herein was made with U.S. Government support under Contract No. H98230-07-C-0383 awarded by the U.S. Department of Defense. The Government has certain rights in this invention. 
    
    
     BACKGROUND OF THE INVENTION 
     The present invention relates generally to operating systems and highly available fault tolerant systems. More particularly, this invention relates to application frameworks deploying applications with dataflow graphs on networks containing circuit switches where such switches are controlled by the application framework. 
     Evolving data workloads, such as stream processing workflows, require new types of interconnects because traditional packet switching is expensive in terms of energy use and is not generally flexible enough to dynamically adapt routing to the data requirements. Electrical packet switches at the center of compute clusters are hard to scale at high data rates, and high port counts require large numbers of parallel switch chips and multi-stage architectures. The need for lower-cost and more flexible interconnects is particularly important in High Performance Computing (HPC) systems. Electrical Circuit Switches (ECS) and Optical Circuit Switches (OCS) have been generally used in reconfigurable networks by the telecommunication industry for statically configured network routes. Both ECS and OCS may provide better power efficiency and allow lower latency through the switch than packet switches. The OCS switch is able to set up a dedicated connection dynamically between any pair of input and output ports. Each port is exclusively used in one connection, and, once configured, the network packets do not have to be inspected by the switch as they pass through. Different connections schemes between input/output pairs creates a unique network topology for the nodes that connect to each switch port, where nodes are typically the network endpoints, normally doing computational tasks on data in the workloads. 
     Stream processing applications are typically long running. In distributed streaming systems, such as that described in a paper entitled,  Adaptive Control of Extreme - scale Stream Processing System , L. Amini, N. Jain, A. Sehgal, J Silber, and O. Verscheure (November 2005), streaming applications are deployed in the system and stream connections are established at runtime. The processing elements (PE) in the applications may, depending on their purpose, become CPU intensive, bandwidth intensive, and/or memory intensive. The streaming system determines the placement of these PEs based on available resources and the requirements of the PEs. The resource availability and requirements, however, can change over time. The node on which a PE is placed may not be suitable or optimal for the PE after a time. The system reacts by moving the PE to another node based on the current resource state. This may not be sufficient in some cases, including when a PE cannot be moved for some reason, or when moving the PE has more associated costs than the system is able to accept. 
     Circuit switched networks not only provide higher bandwidth between nodes but also the flexibility to reconfigure network resources instead of moving PEs. Traditionally, circuit switching technology has been used in wide area networks to provide a dedicated bandwidth connection between a source and destination. 
     Optical circuit switch (OCS) networks may offer a more scalable alternative to cluster interconnection networks. Benefits of OCS include transparency to data rates and protocols, low power consumption, compatibility with wavelength division multiplexing, and the fact that no need for optical-to-electrical and electrical-to-optical conversions are required. 
     The optical switch connects the optical fiber ports on the electronic packet switch of each cluster. Switching fibers connecting to different cluster switch ports form a unique inter-cluster all optical network. Optical switches may also be connected directly onto a node in a cluster for finer granularity. 
     Recent HPC literature has described network topologies that are amenable to evolving data workloads, such as stream processing (see, for example, a paper entitled,  On the Feasibility of Optical Circuit Switching for High Performance Computing Systems , Barker, K. J.; Benner, A.; Hoare, R.; Hoisie, A.; Jones, A. K.; Kerbyson, D. K.; Li, D.; Melhem, R.; Rajamony, R.; Schenfeld, E.; Shao, S.; Stunkel, C.; Walker, P. Supercomputing, 2005). Proceedings of the ACM/IEEE SC 2005 Conference describe a network topology that combines the flexibility and other advantages of Optical Circuit Switches described above, with an aggregation scheme whereby several nodes are connected to one of several Electrical Packet Switches, and each Electrical Packet Switch is connected to the Optical Circuit Switch. The Optical Circuit Switch can be dynamically reconfigured as bandwidth needs evolve to connect one or more ports of each Electrical Packet Switch to one or more ports on other Electrical Packet Switches. In this way, data can be sent from a node, aggregated with traffic from other nodes at their packet switch, routed through the Optical Circuit Switch (which is dynamically configured to connect packets switches together needing the highest bandwidth), routed through the target Electrical Packet Switch, and then demultiplexed to target nodes. However, the technique described in the above-referenced paper for routing through the reconfigurable network to maximize network utilization is dependent on the development of new communication protocols. There is no method for existing network protocols, such as TCP/IP. 
     U.S. Pat. No. 6,792,174B1, entitled “METHOD AND APPARATUS FOR SIGNALING BETWEEN AN OPTICAL CROSS-CONNECT SWITCH AND ATTACHED NETWORK EQUIPMENT,” describes a method for using and controlling Optical Switches in a network. Another patent application, entitled “DUAL NETWORK TYPES SOLUTION FOR COMPUTER INTERCONNECTS,” U.S. Patent Publication No. 2008/0025288, filed on Jul. 27, 2006, describes the use of different network types including circuit switches and packet switches together in a system connecting a set of clusters and nodes. To date, however, there is no known solution for controlling circuit switches in a network, by application frameworks (that are running in the network) and utilizing them for running applications. 
     U.S. Pat. No. 6,671,254B1, entitled “COMMUNICATION NETWORK AND COMMUNICATION NODE USED IN SUCH NETWORK,” and which issued on Dec. 30, 2003, describes a network comprising nodes with an optical cross connect and a packet switch on each node, together with network management based on traffic monitoring. However, there is no application framework for network management, which is specifically based on hardware packet flow monitoring executed on each node. 
     SUMMARY OF THE INVENTION 
     Exemplary embodiments of the invention include a system for implementing stream processing. The system includes an application framework and applications containing dataflow graphs managed by the application framework running on a first network. The system also includes at least one circuit switch in the first network having a configuration that is controlled by the application framework, a plurality of processing nodes interconnected by the first network over one of wireline and wireless links, and a second network for providing at least one of control and additional data transfer over the first network. The application framework reconfigures circuit switches in response to monitoring aspects of the applications and the first network 
     Additional exemplary embodiments include a method for implementing stream processing. The method includes managing a configuration at least one circuit switch in a first network, the configuration managed by an application framework. The application framework and applications contain dataflow graphs that are managed by the application framework running on a first network. The method also includes receiving, by a second network, at least one of control and additional data transfer over the first network. The first network includes processing nodes interconnected by the first network over one of wireline and wireless links. The method further includes monitoring aspects of the applications and the first network, and reconfiguring circuit switches by the application framework in response to the monitoring. 
     Further exemplary embodiments include a computer program product for implementing stream processing. The computer program product includes a storage medium readable by a computer processor and storing instructions for execution by the computer processor for implementing a method. The method includes managing a configuration at least one circuit switch in a first network, the configuration managed by an application framework. The application framework and applications contain dataflow graphs that are managed by the application framework running on a first network. The method also includes receiving, by a second network, at least one of control and additional data transfer over the first network. The first network includes processing nodes interconnected by the first network over one of wireline and wireless links. The method further includes monitoring aspects of the applications and the first network, and reconfiguring circuit switches by the application framework in response to the monitoring. 
     Other systems, methods, and/or computer program products according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such additional systems, methods, and/or computer program products be included within this description, be within the scope of the present invention, and be protected by the accompanying claims. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which: 
         FIG. 1  is a block diagram of a stream processing computer network implementing a reconfigurable optical circuit switch in an exemplary embodiment; 
         FIG. 2  is a block diagram illustrating a stream processing computer network implementing multiple optical circuit switches that are interconnected in accordance with an exemplary embodiment; 
         FIG. 3  is a block diagram illustrating a stream processing computer network implementing multiple optical circuit switches that are interconnected via a dedicated optical switch in an exemplary embodiment; 
         FIG. 4  is a block diagram of a stream processing computer network illustrating connections from an optical circuit switch to processing nodes; 
         FIG. 5  is a flow diagram describing a process for implementing stream processing via reconfigurable optical circuit switch in accordance with an exemplary embodiment; and 
         FIG. 6  is a block diagram illustrating logical components of the stream processing computer network in accordance with an exemplary embodiment. 
     
    
    
     The detailed description explains the exemplary embodiments of the invention, together with advantages and features, by way of example with reference to the drawings. 
     DETAILED DESCRIPTION 
     A stream processing computer network implementing a reconfigurable optical circuit switch, and method therefore, are provided in accordance with exemplary embodiments of the invention. The stream processing computer system controls one or more circuit switches that are present in the network. The streaming system receives input about various aspects of the applications running in the network and about the status of the network itself. Using this information, the system determines what type of network configuration suits the running applications, as well as the placements of the processing elements (PEs) in those applications. 
     The stream processing computer system continuously or periodically reviews the application location allocations as well as the network configuration. Whenever new work arrives or the nature of the resources changes (e.g., new bottlenecks, changing stream requirements, and so on), the system adjusts by reconfiguring the network and relocating PEs to achieve better performance. If multiple network choices are available (such as multiple circuit switched and/or packet switched network components, e.g., switches, nodes), the system can choose one or more of these networks (and components) to route the streams. The choice of network/components and its configuration can be made at runtime. 
     Turning now to  FIG. 1 , an exemplary stream processing computer network system  100  contains one or more optical circuit switches  102 . A secondary network  110 , which may or may not be reconfigurable, is provided so that control data can be routed to all switches (e.g., switches  102 ) and nodes (e.g., nodes  104 ) in the network  100 . This secondary network  110  may or may not be used to carry information traffic. The optical circuit switches  102  may either be connected directly to the processing nodes  104  or indirectly via an intermediate switch (e.g., intermediate switch  103 ). At least one of the nodes  104  is used to host applications processing high bandwidth streaming data. One or more nodes  104  can be connected to the OCS  102  by the same intermediate switch  103 . Several such switches  103  may be used to connect groups of nodes  104  to the OCS  102  system. Each switch  103  may be connected to the OCS  102  with one or more optical links  106 . The links  106  between the intermediate switch  103  and the OCS  102 , as well as those between the OCS  102  and nodes  104 , may be wireline, wireless, or a combination thereof. 
     It should also be noted that the OCS  102  system may or may not connect directly or indirectly to all processing nodes  104  in the network  100 . Rather, an exemplary application framework discovers which nodes  104  have the OCS capability and can use these nodes appropriately. For purposes of illustration, all nodes  104  in the network  100  are shown to be directly or indirectly connected to the OCS  102 . The application framework manages applications running on the network  100 , which contain e.g., data flow graphs. 
       FIGS. 2-4  illustrate alternative embodiments of the network  100  of  FIG. 1 . Components illustrated in  FIGS. 2-4 , which are substantially similar to those described in  FIG. 1  will not be described in further detail. It will be understood that the illustrative embodiments depicted in  FIGS. 2-4  are a small representative of the many types of configurations possible and are not to be construed as limiting in scope. 
     Turning now to  FIG. 2 , an alternative exemplary stream processing computer network system  200  contains one or more optical circuit switches  202 . The switches  202  may be used independently to connect the same or different sets of nodes  204 . Connections  208  may exist in between the switches  202  themselves, which in turn, allow one or more nodes  204  connected to one OCS  202  to connect to one or more nodes  204  on another OCS  202 . Direct/indirect connections (e.g., optical/electrical connections  206  and/or intermediate switches  203 ) connect the switches  202  to the nodes  204 . 
     Turning now to  FIG. 3 , a further embodiment illustrates a network system  300  having switches  302  connected in a different fashion. One or more optical circuit switches  302  are connected to each other via another optical circuit switch  305 . Any number of optical links  308  may be used to connect the optical circuit switches  302 / 305  to one another. The switches  302  may be connected to nodes  304  via direct/indirect connections  303 / 306  as described above with reference to  FIGS. 1 and 2 . 
       FIG. 4  illustrates a stream processing computer network  400  in which two different means by which an optical switch  402  may be connected to nodes  404  are provided. One way is to have an electronic packet switch (EPS) or electronic circuit switch (ECS)  420  with optical transceivers  430  connected to the OCS  402  using fiber optical connections  410 . The EPS/ECS is connected to one or more processing nodes  404  in a cluster  440  of nodes  404 . The switch  420  may have one or more optical links  410  connected to one OCS  402 . Another method is to connect the switch  402  directly to a node  404  with optical transceivers  430  using fiber optical connections  410 . 
     Turning now to  FIG. 5 , an exemplary process (also referred to herein as the application framework) for implementing stream processing via a reconfigurable optical circuit switch network (e.g., one of networks  100 ,  200 ,  300 , and  400 ) will now be described in an exemplary embodiment. The process may be implemented, e.g., via one or more components operating on one or more of the nodes  104 ,  204 ,  304 , and  404 . A sample system including logical components for implementing the exemplary embodiments is shown and described in  FIG. 6 . The nodes may include processors that implement logic for performing the processes described herein. The process continually or periodically gathers network topology and node information in block  502 . This information may be static or dynamic in nature. Static information includes, but is not limited to, data on the number of processors on the node, memory, and the non-reconfigurable network topology. Dynamic information includes, but is not limited to, the CPU load and memory usage in block  504 . Information on network connections, latency, power consumptions, etc., may also be gathered. A small sampling of the attributes of each node that are tracked by the framework includes CPU information, memory information, and interface information (e.g., bandwidth information, drivers, connection information, address, and latency). 
     The framework also tracks the application resource usage information in block  506 . This information includes data on the amount of CPU cycles each PE is using and the bandwidth consumed by each stream that PEs generate or consume (data flow rates). At block  508 , resource requirements of new applications recently submitted into the system, and current applications already running, are determined. 
     Based on all gathered information, the framework optimizer, or scheduler, determines the best application placement and switch configurations in blocks  510  and  512 . The optimization result provides information, such as: PE to node allocation, stream routing information, and reconfigured network topology information (e.g., cluster to cluster connections, cluster to node connections, node to node connections, etc.). New node allocations are provided for each PE. New network topology information is provided for the reconfigurable network(s). For each stream in the system, the route that the stream takes through the network is also provided. 
     Applications are moved from their initial location to their designated nodes in block  514 . Applications that are already running and are not being relocated may need to have their routes reconfigured. Streams that have no direct OCS route from source to destination may take multi-hop routes. This means that, there may be one or more ECS/EPS between the source and destination. This is because when switching from one network configuration to another, routes of several existing applications may also be affected. The system can handle this in several ways. A temporary connection may be set up for short period of time to handle the streams, or the streams may be moved temporarily to a non-reconfigurable network. These details are orthogonal to the present invention. 
     The switch reconfiguration and stream routing changes can take place in parallel (with some degree of synchronization). Once the new routes and topology are in effect, functional, and the system can begin the entire process again from block  514  back to block  502 . 
     Turning now to  FIG. 6 , a block diagram illustrating logical components of the stream processing computer network will now be described in accordance with an exemplary embodiment. Some components of the system of  FIG. 6  may be distributed (not shown) and present on each node. They are responsible for monitoring and controlling the applications on the nodes. One or more instances of the system shown in  FIG. 6  can be used to monitor multiple instances of the distributed components. The system of  FIG. 6  may be described as a combination of three parts. One part comprises monitors  604 ,  606 , and resource information gathering  608 . A second part comprises hardware reconfigurators  610 , application reconfigurators  612 , and optimization response handler  614 . A third part comprises a scheduler  616 . Components  604 ,  606 , and  608  collectively monitor, gather and analyze information on nodes, switches, clusters, applications running, etc. ( 602 ) as described in  FIG. 5 . The components  604 ,  606 , and  608  provide this information to scheduler  616 . The scheduler  616  is responsible for analyzing current system information and determining new parameters for reconfiguring various aspects of the system. It can move applications and reconfigure the network for optimal resource utilization. Components  610 ,  612 , and  614  are collectively responsible for translating scheduler provided output (e.g., new application location and network topology information) into actual commands/operations that are responsible for reconfiguration. The system of  FIG. 6  runs on one or more nodes (e.g., nodes  23 A- 23 D in  FIG. 2 ) in the network that it is managing. Some control information may also be fed back from the reconfiguration part (i.e., components  610 ,  612 , and  614 ) to the monitoring part (i.e., components  604 ,  606 , and  608 ) as represented by a feedback arrow illustrated in  FIG. 6 . 
     The system of  FIG. 6  may operate over various types of networks (e.g., a commonly assigned patent application Ser. No. 12/052,836, entitled, “METHOD, APPARATUS, AND COMPUTER PROGRAM PRODUCT FOR SCHEDULING WORK IN A STREAM-ORIENTED COMPUTER SYSTEM WITH CONFIGURABLE NETWORKS,” filed concurrently herewith, the contents of which are incorporated by reference herein in its entirety. This disclosure summarizes one embodiment of a stream processing system that controls reconfigurable switches. 
     As described above, embodiments can be embodied in the form of computer-implemented processes and apparatuses for practicing those processes. In exemplary embodiments, the invention is embodied in computer program code executed by one or more network elements. Embodiments include computer program code containing instructions embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other computer-readable storage medium, wherein, when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing the invention. Embodiments include computer program code, for example, whether stored in a storage medium, loaded into and/or executed by a computer, or transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via electromagnetic radiation, wherein, when the computer program code is loaded into and executed by a computer, the computer becomes an apparatus for practicing the invention. When implemented on a general-purpose microprocessor, the computer program code segments configure the microprocessor to create specific logic circuits. 
     While the invention has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed as the best mode contemplated for carrying out this invention, but that the invention will include all embodiments falling within the scope of the appended claims. Moreover, the use of the terms first, second, etc. do not denote any order or importance, but rather the terms first, second, etc. are used to distinguish one element from another. Furthermore, the use of the terms a, an, etc. do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced item.