Abstract:
Embodiments of the invention are directed to a system and method for providing a high-volume transaction queueing, reserve ecommerce solution that automatically engages and queues transactions when a primary back-end transaction processing system becomes unresponsive or unstable. Through machine learning algorithms, embodiments of the invention control transaction submission rates by queuing them and throttling the rate at which they are processed based on self-awareness and constant monitoring, feedback and health checks of the primary system. When metrics indicate that the third-party system can begin accepting transactions again, the system automatically feeds the queued transactions along with real-time orders at a rate that the third-party system can successfully manage.

Description:
RELATED APPLICATIONS 
       [0001]    This application claims the benefit of U.S. Provisional Application No. 62/335,740 filed 13 May 2016, entitled “High Volume Transaction Queueing with Machine Learning,” which is incorporated herein by reference. 
     
    
     FIELD OF THE INVENTION 
       [0002]    The present disclosure relates generally to the handling of high volumes of traffic over the internet. More particularly, this disclosure relates to monitoring and measuring the response of a primary ecommerce platform by an external ecommerce system using machine learning to redirect transaction requests to the external system when the primary platform has stability or performance issues. 
       BACKGROUND 
       [0003]    Occasionally, novel products are introduced into the marketplace with their demand far exceeding their supply. Take, for example, Apple&#39;s iPad and iPad2. These items were launched to crowds of consumers waiting outside stores for hours to purchase a limited supply of products. While the seller (i.e. manufacturer, retailer, etc.) may be overjoyed at the demand, such a situation does not create a good customer experience. Purchasers may stand in line for hours only to be told when they reach the counter that there are no products left for them. Anger flares when some consumers purchase multiple quantities for resale on internet auction sites or in foreign countries where the items are not available via legitimate markets, and leave those waiting in line behind them with nothing. 
         [0004]    An ecommerce solution solves some of these problems. Rather than wait in line for hours with uncertain results, consumers may access an online store to purchase the item. The purchaser could even preorder the item in order to receive it when it hits the market. However, business, technical and functional challenges still exist, especially if the ecommerce system is not prepared to handle unusually high demand. 
         [0005]    Consider the problems encountered by online retailer eToys in 1999. eToys experienced an unanticipated high volume of demand for their products during the Christmas season. Items that were out of stock were listed as available and orders were late or not delivered at all. Information systems and their use by customer service representatives were inadequate for providing customers with information about their orders, including product tracking and delivery. All of these issues made order fulfillment extremely expensive for eToys, their stock value plunged and they went bankrupt. While eToys was eventually able to recover, the hard lesson was that information systems have to be up to the task at hand. 
         [0006]    Further, limited supply means that even items offered for purchase online must be allocated to purchasers in a first come first serve manner. Catalogs and inventories must be maintained. Although customers are not physically standing in line, they are in a real, electronic queue. An electronic queue may be even more congested because of the centralized nature of ecommerce purchasing. This may result in connectivity or communication problems if the system is not sufficiently robust. Limits may be placed on the quantity purchased, so the system must ensure that the purchaser not lose his place in queue or risk losing the item. The purchaser may be left not knowing whether the request has been allocated inventory from the limited supply. Finally, once stock is depleted, the seller may lose any information on the potential purchaser, who may or may not be willing to come back when more stock is available. 
         [0007]    High volume, high demand and limited supply orders aside, an online store may experience instability or performance issues for any number of technical reasons, and it may happen at the worst of times, such as during holiday shopping periods. Performance issues and inventory issues leave customers so frustrated they may abandon their online cart and go elsewhere for the product. An external commerce system with high transaction volume queueing (HVTQ) capability prevents this scenario by automatically stepping in to queue orders for later submission to the primary platform while offering the customer a satisfying transaction experience. 
       SUMMARY 
       [0008]    HVTQ is a reserve ecommerce solution that automatically engages and queues orders when a primary back-end transaction processing system becomes unresponsive or unstable. Through algorithms such as those described herein, embodiments of the invention apply machine learning techniques to control transaction submission rates by queuing orders and throttling the rate at which they are processed based on self-awareness and constant monitoring, feedback and health checks of the primary system. When metrics indicate that the third-party system can begin accepting orders again, HVTQ automatically feeds the queued orders along with real-time orders at a rate that the third-party system can successfully manage. 
         [0009]    Many ecommerce platforms have the ability to queue transactions. These queuing solutions notify shoppers after holding the order for a day or two that the order has failed or is successfully placed. The shopper does not know, at the time the order is placed, whether the order will be successful or not. HVTQ was designed to maintain a high-quality user experience. Upon placing an order when HVTQ is running, the user receives an order confirmation and an email stating that their order was received and is being processed. Further, once the order is originally placed, inventory is allocated against that order even though it is in a queued state to ensure successful completion of the order. 
         [0010]    The HVTQ system and method described herein is self-aware. Using algorithms, it constantly monitors the health of the third-party back end system. When system performance degrades to a certain threshold, HVTQ automatically engages (no human intervention is required) and queues orders. Because HVTQ constantly monitors the health of the third-party system, it knows when and at what rate it can begin feeding queued orders back to the third-party system. HVTQ also has an “intermediate” state where it feeds both queued and real-time orders simultaneously to the third-party system. Again, because it is constantly checking the third-party system&#39;s health, it knows the velocity at which it can successfully send orders for processing. This HVTQ solution is innovative because it is system agnostic. It can be integrated into any third-party processing system, giving it ultimate flexibility. 
         [0011]    The HVTQ solution ensures that ecommerce orders are not lost due to degraded system performance or instability. If a transaction processing system is not performant, orders are not successfully placed, and sales are not captured. HVTQ functions almost like an insurance policy—it&#39;s there to take over as soon as a third-party transaction processing system goes down, and ensures that sales are captured and the shopper has the same high quality experience they have when all systems are performing normally. 
         [0012]    Ecommerce platforms have a need for a system and method that can measure the performance of the system and step in to capture transactions when the ecommerce system experiences instability or poor performance. The solution described herein provides that system and method and offers other improvements over the prior art. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0013]    Having described embodiments of the invention in general terms, reference will now be made to the accompanying drawings, wherein: 
           [0014]      FIG. 1  illustrates an exemplary environment in which embodiments described herein may be practiced. 
           [0015]      FIG. 2  illustrates the architecture and data flow of a request made to a shopper node in accordance with an embodiment of the invention. 
           [0016]      FIG. 3  illustrates the architecture and data flow of a request made to a util node in accordance with an embodiment of the invention. 
           [0017]      FIG. 4  provides a state diagram illustrating an exemplary state transition flow for determining when an embodiment of an HVTQ will queue transactions or process them normally. 
       
    
    
     DETAILED DESCRIPTION 
       [0018]    Embodiments of the present invention may be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, embodiments of the invention are shown. The invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the disclosure may enable one of ordinary skill in the art to make and use the invention. Like numbers refer to like components or elements throughout the specification and drawings. 
         [0019]    Embodiments of a high-volume transaction queueing with machine learning system and method (also referred to herein as “external system”) provide an external backup to a primary ecommerce system, (also referred to as a “third-party system,” “primary system” or “primary platform”), when the primary system experiences system overload or performance and stability issues, preventing loss of revenue and providing a satisfying customer experience. This situation frequently arises when a new item is offered for sale, or an item is of particularly high demand, or during expected periods of high internet traffic to a merchant site, such as is experienced during the holidays. 
         [0020]    Embodiments of the invention are designed to monitor the health of the primary (merchant) system by transaction type (e.g. submit shopping cart, a request to calculate tax, or some other type of request). For the purpose of this document, the terms “transaction,” “request,” “call,” “message,” and “order” are synonymous and may be used interchangeably. Other embodiments may protect other calls processed by the primary system using the same system and methods and although they are not described in detail here, the solution is the same as is applied to the order call. 
         [0021]    In some environments operating embodiments of the system, some type of transaction processing takes place on the external system before it is forwarded to the primary system. In other words, the primary system may be integrated with the external system to capture a transaction, perform fraud checks, allocate inventory, etc., before passing an enhanced request on to the primary platform. The external system collects data regarding its attempts to feed transactions to the primary system and, when it determines that the primary system is in distress, it begins queueing transactions. Transactions are queued until the system determines that the primary system is ready to receive new transactions and slowly begins to forward them to the primary system. The queued transactions may be fed to the primary system gradually to avoid overloading it again. 
         [0022]      FIG. 1  illustrates an exemplary environment in which embodiments of this invention operate. In this environment, a transaction originator or user  102  on a computing device might engage with a webstore  106  through a communication network, such as the internet  104 . Behind the webstore there may be a number of services, providers and machinery used to facilitate the web site and the transaction process. For example, the webstore owned by a merchant  108  may be hosted by the merchant (primary platform) or by a full-service, global ecommerce provider  110 , a combination of the two, or in some other configuration. A global ecommerce provider  110  might have multiple data centers located strategically throughout the world; each data center comprising multiple servers, each server comprising at least one processor, a non-transitory memory device for data storage and modules and applications, and communications device. Some data centers may use a co-located pod configuration  112 , whereby numerous order taker application servers  114 , each with no administrative capability, but with its own order taker database, exist to capture transactions. The transactions are then transferred to an operational data store (ODS)  116  for full order processing. Within each order taker  114 , transaction processing application modules are stored, and called to perform specific functions on the data submitted with a transaction. Each pod is given a name so transactions may be directed to a specific pod, if desired. 
         [0023]    There are many benefits to this type of system. The order takers  114  can continue to accept transactions while maintenance is done on the ODS  116 . Communications between the order taker and the ODS  116  may be two-way—maintenance data changes (look-up data, e.g. site, product, catalog) may be pushed from the ODS  116  to the order taker using a data replication tool. The concept of shared data exists in this type of system as well, by including a shared database. Shared data is data for which there can be only one copy that must be visible to all applications instances at all times. Examples of shared data include originator/user data and Digital Rights. This type of configuration provides parallel, redundant, executing applications. The ODS  116  may provide all of the backend services required for transaction processing, or some may be provided by the ODS  116  and some by the primary platform  108 . Each may complete transaction processing and then forward a request for fulfillment to fulfillment center systems  118 , such as a physical goods warehouse or a digital downloads center. 
         [0024]    In at least one exemplary integration, a web merchant  108  may prefer to perform much of the transaction processing itself, while contracting with a global ecommerce services provider  110  for any number of front-end or back-end services. Front-end services might be related to fraud screening, inventory allotment, for example, and back-end services might be related to tax calculations, payment processing, and other services. The global ecommerce services provider  110  performs the contracted services and forwards a message containing the user request, enhanced with data related to the services it has processed, to the primary system  108 . 
         [0025]    Web sites and services, being machines with limited resources, may experience instability when a high volume of transactions is submitted during a very short period of time, as can happen with high demand products sold during the holidays. During periods of peak traffic on a node (e.g. web server, application server, etc), the ecommerce service provider  110  must be able to continue to accept orders and provide the transaction originator with an acceptable experience while preventing further instability of the primary selling platform  108 . When this occurs, the ecommerce system  110  must hold incoming orders until the primary system  108  has regained stability, and then continue to monitor system health. 
         [0026]    In some embodiments, a shopper node (order taker server  114  as described above) processes the transaction and submits it to the primary platform  108  while monitoring the health of the primary system  108  to determine a course of action for an incoming transaction. A service monitoring module residing on the shopper node  114  or util node (an ODS server  116  capturing and reprocessing queued transactions) may comprise computer code, which when executed by the processor collects data on each call made to the primary system, calculates metrics and sets or resets a circuit breaker that directs transactions to queue in times of primary system  108  distress. Processes resident on the util node, discussed below, resubmits queued transactions for processing. As transactions are presented to the primary platform  108 , data is collected regarding the health of the primary system  108  and is used to control the flow of transactions to the primary platform  108 . Similarly, when an attempt is made to resubmit queued orders, a service monitoring module on the util node  116  collects data and calculates metrics that set the circuit breaker on that node. 
         [0027]    Nodes (processing locations) comprise computing devices, such as the user computing device, and the servers hosting the ecommerce systems (global ecommerce and primary platform) and HVTQ modules used to implement embodiments of the invention, may include a communication device, a processing device and a memory device. The processing device is operatively coupled to the communication device and the memory device. As used herein, “processing device” generally includes circuitry used for implementing the communication and/or logic functions of the particular system. For example, a processing device may include a digital signal processor device, a microprocessor device, and various analog-to-digital converters, digital to analog converters and other support circuits alone or in combination. Control and signal processing functions of the system are allocated between these processing devices according to their respective capabilities. The processing device may include functionality to operate one or more software programs based on computer readable instructions which may be stored in a non-transitory memory device, typically as modules encapsulating code to implement a particular functionality. The processing device uses the communication device to communicate with the network and the devices and systems on the network, such as, but not limited to the user computing device, the global ecommerce (external) system, and the primary platform. As such, the communication device generally comprises a modem, server or other device for communicating with other devices on the network. 
         [0028]      FIGS. 2 and 3  illustrate the architecture of a high-volume transaction queueing system.  FIG. 2  illustrates a shopper node (order taker)  114 . Processes reside on the shopper node to (1) receive and pre-process transactions; (2) send the transactions, enhanced with data derived from pre-processing to the primary platform  206 ; (3) collect data regarding performance of the transaction submission  216  to the primary platform; (4) calculate indicia of performance used to determine whether incoming orders should be processed normally or queued (and setting the circuit breaker to closed or open  204 , depending on that determination; (5) set order status as either “submitted” or “queued” and store those transactions in an order taker data store  114 ; (6) transfer orders  214  to the ODS  116 ; and (7) transfer metrics  218  to a metrics system  318 . 
         [0029]    Referring again to  FIG. 2 , the user  102  may submit an order to the shopper node  110 ,  114 . The shopper node receives the transaction and performs its front-end processing tasks, such as inventory checking and fraud analysis, then checks  202  the circuit breaker  204  to determine whether the primary system  108  can receive transactions. If the circuit breaker  204  is closed, the order is submitted  206  to the primary platform  108  for processing. The service monitoring module  218  collects data (described in more detail below) regarding this message flow, for example, a timestamp for when the message was sent and when a response  208  was received. The data is used to determine the configurable settings on the circuit breaker  204 , including volumeThreshold, errorThreshold, sleepWindow and rateLimit. If the transaction receives a response indicating that the transaction has been submitted without error or delay, the order state is set to ‘submitted’  210 . 
         [0030]    When the service monitoring module  218  determines that the primary system  108  is unstable or in distress, the circuit breaker  202 ,  204  is closed. As long as the circuit breaker  202 ,  204  is closed, transactions are pre-processed normally, but instead of being submitted to the primary platform, their status is set to ‘queued’  212  and the transaction requests are held in queue. All transactions are then stored in the shopper node database  114  as ‘submitted’ or ‘queued’ and will be transferred to the ODS ( FIG. 4 ) via an order transfer service  216  on a routine basis. Queued order metrics from the service monitoring module  218  are also stored in the shopper node and are transferred to a global metrics  220  system on a routine basis. 
         [0031]      FIG. 3  illustrates a util node  116  on an ODS. Util nodes provide utility type services to the ODS in a pod, including submitting queued transactions to the primary platform. Processes reside on the util node  116  to (1) receive transactions from the pod order takers  114 ; (2) receive metrics from the order takers  114  which were collected as the order taker interacted with the primary platform, and which regard the health of the primary platform; (3) identify, select, collect and prepare  302  queued transaction records  214  from the ODS  116 ; (4) create a real time job for submitting queued transactions  304 ; (5) collect data regarding performance of the queued order re-submission  316  to the primary platform; (6) calculate indicia of performance used to determine whether the queued transactions should be resubmitted or continue to be held in queue; (7) reset transaction status as either “submitted  312 ” or leave as “queued  314 ” and store those transactions in an ODS data store  116 ; and (8) update metrics  218  in a metrics monitoring system  330 . 
         [0032]    Referring again to  FIG. 3 , once transferred  216  to the ODS util node  116 , the queued orders are pulled from the ODS  116  by a queued order processor  302 . A real-time job  304  is created for resubmitting the orders to the primary platform  108 . A service monitoring module  218 , which has collected data from previous attempts to resubmit queued orders and continues to collect data as queued transaction resubmission is attempted or completed, sets the circuit breaker  306  to open or closed. If the circuit breaker  306  is closed, the orders are submitted to the primary platform  108  for processing  206  and data is collected regarding the response  316  received from the primary platform  108 . If the orders are successfully submitted to the primary platform  108 , their status is set to ‘submitted’  312 . If the circuit breaker  306  is open, the orders will continue to be deferred to the queued state  314  until the primary system shows signs of recovery. If a response  316  from resubmission continues to indicate performance issues, there is no change in the status of the queued order  314 . Once the system shows signs of recovery, the queued orders are resubmitted  304  at a rate limited pace determined by the metrics system to avoid causing further instability with the primary platform  108 . During this time, new transactions  206  may be submitted along with queued transactions  304  at a rate that monitoring has indicated the system can process normally without causing further performance issues. 
       Machine Learning and Circuit Breaker Transitions 
       [0033]    As described above, the service monitoring module aggregates data related to the health of the primary platform in order to determine the value of parameters used in setting the circuit breaker, which determines whether the transactions are processed normally or are queued to be retried when the primary platform is healthy again. Each shopper node may have one of three states; each util node one of two, as are listed in Table 1. 
         [0000]    
       
         
               
             
               
               
               
             
           
               
                 TABLE 1 
               
             
             
               
                   
               
               
                 Available States for Each Node 
               
             
          
           
               
                   
                 Node 
                 Description 
               
               
                   
                   
               
               
                   
                 Util normal 
                 util nodes retrying queued orders process 
               
               
                   
                   
                 normally 
               
               
                   
                 Util deferred 
                 util nodes not processing queued orders  
               
               
                   
                   
                 (not retrying). 
               
               
                   
                 Shopper Normal 
                 shopper nodes accepting incoming orders  
               
               
                   
                   
                 process normally. 
               
               
                   
                 Shopper Limited 
                 shopper nodes process orders using rate  
               
               
                   
                   
                 limiting.  
               
               
                   
                   
                 Concurrent requests will be queued. 
               
               
                   
                 Shopper Queueing 
                 Shopper nodes queue all traffic. 
               
               
                   
                   
               
             
          
         
       
     
         [0034]    Under normal circumstances, both the util node and the shopper node circuit breakers will be closed. Under high load, if the circuit breakers trip, the size of the queue will grow rapidly. If the queue size reaches a configurable limit (Q1), all circuit breakers on all nodes will be opened. After a configurable cool down period (S) the circuit breakers on the util nodes will be set to a rate-limited state. After this point the health of the system will be evaluated by querying the number of orders that were retried successfully versus the number of orders that were retried and requeued since the nodes were all set to open. If a percentage of these requests greater than a configurable threshold (P) are successful, the util node circuit breakers will be set to closed. If the percentage of successful requests is below the threshold, it indicates that the system is not yet healthy, and the util nodes circuit breakers will be set to open again. Once the util node circuit breakers are in a closed state, they will remain in this state until the size of the queue drops below a configurable size (Q2) at which point the shopper nodes will be set to a rate limited state. If instead the size of the queue increases to above the limit Q1, indicating that the system is once again unhealthy, the util node circuit breaker will be set to open again. When the shopper nodes are in the rate limited state and the queue size drops to below a low configurable size (Q3), the shopper node circuit breaker will return to a closed state. Table 2 describes these configurable and calculated parameters. The specified configurable values are set in a config properties file. 
         [0000]    
       
         
               
             
               
               
               
             
           
               
                 TABLE 2 
               
             
             
               
                   
               
               
                 Node Parameters 
               
             
          
           
               
                 Parameter 
                 Name 
                 Definition 
               
               
                   
               
               
                 S 
                 Sleep window 
                 The time that must have elapsed after starting 
               
               
                   
                   
                 queueing all orders to start retrying queued 
               
               
                   
                   
                 orders. 
               
               
                 P 
                 Successful Percentage Threshold 
                 The minimum percentage of successful retry 
               
               
                   
                   
                 attempts allowed. 
               
               
                 Q1 
                 Start Queueing Percentage 
                 The minimum percentage of successful orders 
               
               
                   
                 Threshold 
                 before all orders will be queued. 
               
               
                 Q2 
                 Half Queueing Threshold 
                 The size the queue must go below to allow 
               
               
                   
                   
                 shopper nodes to process orders using rate 
               
               
                   
                   
                 limiting 
               
               
                 Q3 
                 Stop Queueing Threshold 
                 The size the queue must go below to allow 
               
               
                   
                   
                 shopper nodes to process orders 
               
               
                 W1 
                 Start Queueing Rolling Window 
                 The look back period for failed orders when 
               
               
                   
                   
                 evaluating whether to queue. 
               
               
                 W2 
                 Date Range for Requisition 
                 The look back period for failed orders when 
               
               
                   
                 Retry Job in Hours 
                 evaluating whether to stop queueing. 
               
               
                 W3 
                 Time Interval for Recently 
                 The look back period for queued orders so 
               
               
                   
                 Processing Reqs In Minutes 
                 queued orders are not retried immediately. 
               
               
                   
               
             
          
         
       
     
         [0035]    Data, primarily time values, counts and response results (failures or successful attempts and data about them), is collected and metrics calculated at both a global level and a local level, which allows the system to remove a node from service if it appears to be unstable. Configurable settings may be first estimated using historical knowledge of the number of transactions expected, and then recalculated or re-estimated as production data is received. Table 3 provides exemplary global settings, and Table 4 provides exemplary local settings, both with exemplary values. The values listed in Tables 3 and 4 merely provide examples of the configurable values that may be used. Those of ordinary skill in the art will understand that these values should be set to optimize the system practicing the embodiment. 
         [0000]    
       
         
               
             
               
               
               
             
           
               
                 TABLE 3 
               
             
             
               
                   
               
               
                 Exemplary Global Queueing Threshold Settings 
               
             
          
           
               
                 Property 
                 Value 
                 Description 
               
               
                   
               
               
                 Q1 
                 88% 
                 The minimum number of queued orders (timeouts) to 
               
               
                   
                   
                 trigger a transition to the U_OPEN_S_OPEN state. 
               
               
                 S 
                 30000 ms (5 min) 
                 The time that must have elapsed to reattempt sending 
               
               
                   
                   
                 orders to the client after the last state transition 
               
               
                   
                   
                 into U_OPEN_S_OPEN (Queuing orders everywhere). 
               
               
                 Q2 
                 240 
                 The maximum number of orders in the queue to allow a 
               
               
                   
                   
                 transition to the U_CLOSED_S_HALF queuing strategy 
               
               
                   
                   
                 state from the U_CLOSED_S_OPEN state. 
               
               
                   
                   
                 In half queuing state there may be a small number of 
               
               
                   
                   
                 queued orders left to process (240, for example) but normal 
               
               
                   
                   
                 processing of transactions should begin slowly while 
               
               
                   
                   
                 continuing to queue. Setting to 10% of orders per hour 
               
               
                   
                   
                 (2,400/hr at peak). 
               
               
                 P 
                 95% 
                 Attempt to resubmit to client every 5 minutes after 
               
               
                   
                   
                 queueing begins. Unless 95% of resubmits are successful in 
               
               
                   
                   
                 a cycle of resubmits, move breakers back to the U_OPEN, 
               
               
                   
                   
                 S_OPEN state (transactions may not be processed either in 
               
               
                   
                   
                 ODS (util) or Shopper). 
               
               
                   
                   
                 If 95% or more resubmits are successful but the system is 
               
               
                   
                   
                 still above the halfQueuing threshold (see above), this 
               
               
                   
                   
                 means that the client endpoint is healthy again but there 
               
               
                   
                   
                 may be a large queue of transactions to reprocess. If a large 
               
               
                   
                   
                 queue of transactions is currently being resubmitted to 
               
               
                   
                   
                 client, the system should not start sending Shopper “Submit 
               
               
                   
                   
                 Cart” directly to the client to avoid overwhelming it. 
               
               
                 Q3 
                 200 
                 Queue length is small enough for us to start sending 
               
               
                   
                   
                 Shopper “Submit Cart” directly to Client. 
               
               
                   
               
             
          
         
       
     
         [0000]    
       
         
               
             
               
               
               
             
           
               
                 TABLE 4 
               
             
             
               
                   
               
               
                 Exemplary Local Node Threshold Settings 
               
             
          
           
               
                 Property 
                 Value 
                 Description 
               
               
                   
               
               
                 FR_SHP 
                 50% 
                 FAILURE RATE Shopper nodes: Percentage of requests 
               
               
                   
                   
                 that should fail in RW_SHP time to make “Submit Cart” 
               
               
                   
                   
                 action skip invocation of ProcessCart and queue right way. 
               
               
                   
                   
                 Unless there is an outage, a typical success rate is 99.8%.  
               
               
                   
                   
                 A 50% failure rate should be a clear indicator that a node is  
               
               
                   
                   
                 having trouble submitting orders to the primary platform. 
               
               
                 RW_SHP 
                 30 minutes 
                 ROLLING WINDOW Shopper nodes: Time window to 
               
               
                   
                   
                 collect error/success percentage. 
               
               
                   
                   
                 The size of the window is the time it takes to react to an outage. 
               
               
                   
                   
                 A smaller window will elicit a quicker reaction to an outage  
               
               
                   
                   
                 but may allow enough requests to provide a meaningful 
               
               
                   
                   
                 success/failure percentage. 
               
               
                 VT_SHP 
                  4 
                 VOLUME THRESHOLD Shopper nodes: Minimum number 
               
               
                   
                   
                 of request that should happen in rolling window to decide to 
               
               
                   
                   
                 make the circuit trip and queue orders straight away. 
               
               
                   
                   
                 Setting this to 4 means that if there are 3 requests in the 30  
               
               
                   
                   
                 minute Rolling Window and all fail, the circuit will still not trip. 
               
               
                 SW_SHP 
                 30 minutes 
                 SLEEP WINDOW Shopper Nodes: In past outages things 
               
               
                   
                   
                 typically go bad for 3-4 hours at least. Half an hour sleep 
               
               
                   
                   
                 window seems reasonable. 
               
               
                 FR_BGW 
                 10% 
                 FAILURE RATE Background (util) nodes: At a volume 
               
               
                   
                   
                 threshold of 20 this means that in a 5-minute interval at 
               
               
                   
                   
                 least 18 order reprocesses out of 20 should pass. 
               
               
                 RW_BGW 
                  5 minutes 
                 ROLLING WINDOW Background (util) nodes: on a single  
               
               
                   
                   
                 BGW node there may be between 20 and 25 orders in 
               
               
                   
                   
                 5 minutes; 20 when queuing 100% of 2,400 order/hours 
               
               
                   
                   
                 across 10 BGW nodes. 2.5 when at 12% failure rate. 
               
               
                 VT_BGW 
                 20 
                 VOLUME THRESHOLD Background (util) nodes:  
               
               
                   
                   
                 Keep volume threshold at 100% queuing (full outage). 
               
               
                   
                   
                 Anything less than a full outage, try to send across 
               
               
                   
                   
                 all orders possible. 
               
               
                 SW_BGW 
                  5 minutes 
                 SLEEP WINDOW Background (util) nodes: 
               
               
                   
               
             
          
         
       
     
         [0036]      FIG. 4  illustrates the logic used to set node states and initiate transitions. These are exemplary states and transitions. One of ordinary skill in the art would recognize that the states and properties may be designed and configured to fit any the properties and values required by a particular system. In this example, nodes may be in several state combinations: util normal, shopper normal  402 ; util deferred, shopper queueing  404 ; util normal, shopper queueing  406 ; and util normal shopper limited  408 . Referring to the property codes above, at  408 , the percent of successful total requests is equal to or greater than Q1 for the previous W1 minutes, and the nodes are both in the normal state  402  with circuit breakers opens. When the percent of successful total requests degrades to less than Q1 for the previous W1 minutes, and it has been less than S milliseconds since the last state transition, the shopper node circuit breakers are open and transactions begin queueing; util (resubmitting queued orders) is deferred  404 . At  406 , it has been more than S milliseconds since the last state transition, the percent successful retry requests is greater than P since the last evaluation and queue length is greater than Q2 for the last W2 hours, not including the last W3 minutes. Here, util is normal and the shopper node is queueing. At  408 , the queue length has been greater than Q3 for the previous W2 hours, not including the previous W3 minutes and the percent successful total requests are greater than Q1 for the previous W1 minutes; the percent successful retry requests is greater than P since the last evaluation and queue length is less than Q2 for the last W2 hours not including the previous W3 minutes. Here,  408 , util is normal and shopper is limited. When queue length is less than Q3 for the previous W2 hours and the percent successful requests has been greater than Q1 for the previous W1 minutes, the node states both revert to normal  402 . 
         [0037]    While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that this invention not be limited to the specific constructions and arrangements shown and described, since various other changes, combinations, omissions, modifications and substitutions, in addition to those set forth in the above paragraphs, are possible. Those skilled in the art will appreciate that various adaptations and modifications of the just described embodiments can be configured without departing from the scope and spirit of the invention. Therefore, it is to be understood that, within the scope of the appended claims, the invention may be practiced other than as specifically described herein.