Abstract:
A method includes deriving packet delivery statistics from a User Datagram Protocol (UDP) stream simulating a service level provided by a Voice over Internet Protocol (VoIP) network and transmitted across the VoIP network at a first pre-defined interval, and processing the derived packet delivery statistics at a second pre-defined interval to generate network performance statistics for the VoIP network, where the second pre-defined interval is substantially larger than the first pre-defined interval.

Description:
BACKGROUND  
         [0001]    The invention relates to a Voice over IP (VoIP) network performance monitor.  
           [0002]    Packet-based networks, in particular, VoIP networks, are rapidly emerging as a viable alternative to traditional telephony (that is, circuit switched networks). VoIP is viewed as an attractive option for voice transport in that it allows live voice conversations to be integrated with existing IP data and image applications. To be a truly competitive alternative, VoIP must emulate the performance of traditional telephony and do so using a protocol that was optimized for data traffic. The characteristics of data traffic are quite different from those of voice traffic.  
           [0003]    Unlike data traffic, voice traffic is extremely intolerant of delay and delay variation (or “jitter”), as well as, packet loss. Much work has been done in the area of packet delivery to provide end-to-end Quality of Service (QoS). Service level agreements (SLAs) for VoIP, like those for conventional data IP networks, tend to be based on conventional data network metrics—that is, guaranteed service levels are expressed solely in terms of packet level performance, for example, packet loss, jitter, and round-trip delay.  
           [0004]    Traditional network performance measurement tools use Packet InterNet Gropers (PINGs) to measure packet loss, jitter and round-trip delay. However, VoIP networks are packet-based networks that transmit data in one-way steady streams of User Datagram Protocol (UDP) packets, which PINGing tools cannot simulate. In addition, traditional network monitoring systems accept performance data that has been averaged over an extended period of time, such as a day or a month. While such measurements are acceptable for data transmissions, which are typically TCP in nature, the real-time nature of VoIP service makes daily and/or monthly averaging of performance results inaccurate in representing a VoIP network user&#39;s experience of the service.  
         SUMMARY  
         [0005]    In general, according to one aspect of the invention, a method includes deriving packet delivery statistics from a User Datagram Protocol (UDP) stream simulating a service level provided by a Voice over Internet Protocol (VoIP) network and transmitted across the VoIP network at a first pre-defined interval, and processing the derived packet delivery statistics at a second pre-defined interval to generate network performance statistics for the VoIP network, where the second pre-defined interval is substantially larger than the first pre-defined interval.  
           [0006]    One or more of the following features may also be included.  
           [0007]    A first client terminal connected to the VoIP network generates the UDP stream. In certain embodiments, generating a UDP stream includes selecting a service level and adjusting payload sizes and bit rates of the UDP packets according to the selected service level.  
           [0008]    As yet another feature, the method also includes setting an IP precedence bit to specify a class of service for each UDP packet. The UDP stream is transmitted by a first client terminal connected to the VoIP network to a second client terminal connected to the VoIP network.  
           [0009]    As another feature, the packet delivery statistics are derived from the UDP stream by a first client terminal connected to the VoIP network. Deriving packet delivery statistics includes measuring packet loss of the UDP stream at each first pre-defined interval. In certain embodiments, deriving packet delivery statistics further includes measuring jitter of the UDP stream at each first pre-defined interval.  
           [0010]    As yet another feature, deriving packet delivery statistics includes measuring round-trip delay of the UDP stream at each first pre-defined interval. The network performance statistics are generated by a monitoring device connected the VoIP network.  
           [0011]    The method also includes generating a graphical report showing a distribution of the derived packet delivery statistics over the second pre-defined interval. In certain embodiments, a first client terminal is connected to a first VoIP Point of Presence (VoIP PoP) on the VoIP network and a second client terminal is connected to a second VoIP PoP on the VoIP network and the method also includes triggering a traceroute that traces a route between the first VoIP PoP and the second VoIP PoP if any one of the network performance statistics exceeds a pre-configured threshold.  
           [0012]    According to another aspect of the invention, a Voice over Internet Protocol (VoIP) network performance testing system includes a VoIP network, a first client terminal connected to the VoIP network, the first client terminal configured to generate a stream of User Datagram Protocol (UDP) packets simulating a service level provided by a Voice over Internet Protocol (VoIP) network, transmit the UDP stream across the VoIP network to a second client terminal connected to the VoIP network, derive packet delivery statistics from the UDP stream at a first pre-defined interval, and a monitoring device connected to the VoIP network, the monitoring device configured to process the derived packet delivery statistics at a second pre-defined interval, and generate network performance statistics for the VoIP network, where the second pre-defined interval is substantially larger than the first pre-defined interval.  
           [0013]    One or more of the following features may also be included.  
           [0014]    In certain embodiments, the UDP stream is generated by a first client terminal connected to the VoIP network.  
           [0015]    As another feature, generating includes controlling the first client terminal to select a service level and generate a UDP stream having payload sizes and bit rates corresponding to the selected service level.  
           [0016]    As yet another feature, system also includes controlling the first client terminal to set an IP precedence bit to specify a class of service for each UDP packet.  
           [0017]    As another feature, the system includes controlling the monitoring device to trigger an alarm if any one of the network performance statistics exceeds a pre-configured threshold.  
           [0018]    In certain embodiments, the first client terminal is connected to a first VoIP Point of Presence (VoIP PoP) on the VoIP network and the second client terminal is connected to a second VoIP PoP on the VoIP network, the system also including controlling the monitoring device to trigger a traceroute that traces a route between the first VoIP PoP and the second VoIP PoP if any one of the network performance statistics exceeds a pre-configured threshold.  
           [0019]    The system may also include controlling the first client terminal to store results of the traceroute in a traceroute file, wherein the results comprise Internet Protocol (IP) addresses of each router in the route, and send the traceroute file to the monitoring device.  
           [0020]    Embodiments may have one or more of the following advantages. The invention enables a VoIP-simulated UDP stream to be generated between every possible pair of VoIP Points of Presence (VoIP PoPs) for use in testing the performance of a VoIP network. The invention generates and reports performance results in a manner that reflects the real-time nature of the VoIP services offered by the VoIP network. With these performance results, a network operator can assess the quality of the VoIP network&#39;s services on a daily basis for use with Service Level Agreements, as well as, identify possible problem areas in the VoIP network that may affect performance.  
           [0021]    The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims. 
       
    
    
     DESCRIPTION OF DRAWINGS  
       [0022]    [0022]FIG. 1 is a block diagram illustrating a Voice over Internet Protocol (VoIP) network performance test topology.  
         [0023]    [0023]FIG. 2 is a flowchart of a process for testing the performance of a VoIP network.  
         [0024]    [0024]FIG. 3 shows a sample graphical user interface in accordance with the present invention. 
     
    
       [0025]    Like reference symbols in the various drawings indicate like elements.  
       DETAILED DESCRIPTION  
       [0026]    In one exemplary commercial setting, as shown in FIG. 1, a network infrastructure  100  of a VoIP network  102  is maintained by a VoIP network operator (not shown), such as Genuity Inc. of Woburn, Mass., and made available to that VoIP network operator&#39;s customers (not shown), for example, retail service providers who use the infrastructure and related services of the VoIP network operator to provide VoIP services to end users. In a large-scale operation, the network operator supports a large number of VoIP Points of Presence (VoIP PoPs)  104 , in different geographic regions for coverage of a larger territory, for example, national level coverage. Each VoIP PoP  104  provides a point of entry to and termination from the VoIP network  102 .  
         [0027]    The network operator monitors the performance of the VoIP network  102  through the use of a VoIP-performance monitor server (or performance monitor server)  106  and VoIP-performance monitor clients (performance monitor clients)  108  deployed at VoIP PoPs  104  throughout the network  102 . Each of the performance monitor clients  108  is configured to generate VoIP-simulated UDP streams to others of the performance monitor clients  108  over the VoIP network  102 , and perform network parameter and performance measurements (collectively called “performance data”). Preferably, a VoIP-simulated UDP stream is produced by the performance monitor client  108  for each level of service as determined by the type of codec (i.e., coder/decoder) used by a VoIP communications device that is performing the voice encoding and decoding operations at the VoIP PoP  104 . In the VoIP PoP  104   a  shown in FIG. 1, a gateway  110  is used to implement one or more coding schemes to support voice encoding/decoding between protocols of the VoIP network  102  and a conventional telephony network, such as a Public Switched Telephone Network (PSTN)  112 .  
         [0028]    Types of codecs include, but need not be limited to, the following: waveform codecs, source codecs, and hybrid codecs. With waveform codecs, an incoming voice signal is sampled, coded and the coded samples converted to quantized values, which are used to reconstruct the original voice signal. Waveform codecs (e.g., G. 711 and G. 726) produce high quality sound but consume a significant amount of bandwidth. Source codecs try to match an incoming voice signal to a mathematical model of speech generation. That model is used to reconstruct the original voice signal. The source codec operates at low bit rates but tends to produce poor quality sound. Hybrid codecs (e.g., G. 729 and G. 723) use some amount of waveform matching as well as knowledge of how the original sound was generated. They tend to provide fairly good quality sound at lower bit rates than waveform codecs. Due to its lower bandwidth requirements, G.729 is the codec of choice for VoIP network performance testing.  
         [0029]    [0029]FIG. 2 shows a process  200  implemented by the performance monitor server  106  and the performance monitor clients  108 . The performance monitor client  108   a  is configured to generate and transmit ( 202 ) a VoIP-simulated UDP stream to every VoIP PoP  104  in the VoIP network  102 , including the VoIP PoP  104   b  having a performance monitor client  108   b.  In one implementation, the performance monitor client  108   a  in VoIP PoP  104   a  is configured to automatically start a new 24-hour testing session, say, for example, every morning after midnight, by generating and transmitting a UDP stream to the performance monitor client  108   b  in VoIP PoP  104   b.  The performance monitor client  108   a  collects ( 204 ) samples of raw performance data (e.g., packet loss measurements, jitter measurements, and round-trip delay measurements) on the incoming UDP stream at pre-defined intervals, for example, 10-second intervals. Thus, each sample contains 10-seconds worth of data.  
         [0030]    The performance monitor server  106  polls ( 206 ) the performance monitor client  108   a  at pre-defined intervals, for example, at 15-minute intervals to gather the samples of raw data. The length of the collection interval and the polling interval may be set at any number that provides a sufficiently large sample size. The performance monitor server  106  can be designed to process ( 208 ) each sample&#39;s raw data to calculate the average, maximum, and minimum packet loss over the 15-minute period. Similar calculations can be made based on the jitter measurements and the round-trip delay measurements. Once processed, the performance monitor server  106  generates a time stamped 15-minute summary report that includes these calculations, as well as, histograms that show the distribution of the packet loss, jitter, and round-trip delay measurements during that particular 15-minute testing window. It should be noted that the data points used for the calculations and the histograms correspond to the 10-second samples. The timestamp indicates both the time and date associated with the “start” and “end” of a 15-minute testing window.  
         [0031]    In one implementation, the performance monitor server  106  examines ( 210 ) the average and maximum values in the 15-minute summary report and triggers ( 212 ) an alarm, such as an e-mail to an employee of the network operator, if an average value or a maximum value exceeds a configurable threshold. In another implementation, the performance monitor server  106  examines ( 214 ) the average and maximum values in the 15-minute summary report and triggers a traceroute between the appropriate VoIP PoP pair (in this case, VoIP PoPs  104   a  and  104   b ) if an average value or maximum value exceeds a configurable threshold, or if the percentage of failure events (defined as high packet loss, high jitter, or high round-trip time in the 10-second sample) during the 15-minute interval exceeds configurable thresholds, such as packet loss which is greater than one percent. These thresholds may be expressed in the format of a Service Level Agreement (SLA) for VoIP services on the VoIP network  102 . For example, if the average packet loss value for a UDP stream transmitted by the performance monitor client  108   a  and received by the performance monitor client  108   b  exceeds a configurable threshold, the performance monitor server  106  sends a signal to the performance monitor client  108   a  to trigger a traceroute that traces a route between the VoIP PoP  104   a  and the VoIP PoP  104   b.  Since UDP provides no connections but simply delivers packets, it is possible that traffic between the VoIP PoPs  104   a  and  104   b  may be sent over more than one route. Over time it is likely that the route between the VoIP PoPs  104   a  and  104   b  will vary due to changes in internet connections, modifications of routers, and changes in service. However, over the short term these changes will usually not be present, and a traceroute is a useful tool to test connectivity between the two VoIP PoPs  104   a  and  104   b.  At the conclusion of the traceroute, the performance monitor client  108   a  stores ( 216 ) the results of the traceroute—that is, the IP addresses of all the routers along the route between the VoIP PoPs  104   a  and  104   b —in a time stamped traceroute file and sends the traceroute file to the performance monitor server  106 . The timestamp indicates what time the traceroute was executed.  
         [0032]    In one implementation, the network operator monitors the performance (e.g., discard rate, error rate) of the routers in the VoIP network  102  through the use of a router performance monitor server  114 . The router performance monitor server  114  may be implemented using commercially available hardware and software. At the conclusion of a traceroute triggered by the performance monitor server  106 , the performance monitor server  106  may be configured to establish a link to the router performance monitor server  114  to obtain the performance statistics (e.g., discard rate, error rate) associated with each traceroute hop&#39;s router, and store the results of the query in the traceroute file.  
         [0033]    The performance monitor server  106  repeats the polling ( 206 ) of the performance monitor client  108   b  at 15-minute intervals until the date associated with the “end” time of the most-recent 15-minute testing window is different from the date associated with the “start” time of the first 15-minute testing window in the 24-hour testing session. When this occurs, the performance monitor server  106  does the following:  
         [0034]    (1) The performance monitor server  106  processes the raw data collected during the most-recent 15-minute testing window, generates a 15-minute summary report, and triggers a traceroute or alarm, if necessary.  
         [0035]    (2) The performance monitor server  106  processes all of the raw data collected during the 24-hour testing session to calculate the average, maximum, and minimum packet loss over the 24-hour period. Similar calculations can be made based on the jitter measurements and the round-trip delay measurements. Once processed, the performance monitor server  106  generates a 24-hour summary report that includes these calculations, as well as, histograms that show the distribution of the packet loss, jitter, and round-trip delay measurements per 10 seconds during that 24-hour testing session. By taking the performance measurements at different times of the day over the 24-hour period, the effect of different system loads on the measured quantity can be seen.  
         [0036]    In one implementation, each performance monitor client  108  connected to the VoIP network  102  is configured to generate and transmit a VoIP-simulated UDP stream tagged with an IP precedence bit. Setting the IP precedence bit allows the performance monitor server  106  to check the performance of VoIP services when, for example, Quality of Service (QoS) options are configured on the VoIP network  102 . In another implementation, each performance monitor client  108  is configured to generate and transmit two VoIP-simulated UDP streams simultaneously—one tagged with an IP precedence bit and one without. The generation and transmission of side-by-side streams permits the performance monitor server  106  to fully compare the relative performance of the VoIP service across different network configurations. In another implementation, each performance monitor client  108  is configured to generate and transmit multiple VoIP-simulated UDP streams, each UDP stream simulating a different voice codec used by the VoIP service.  
         [0037]    A web server  116  connected to the performance monitor server  106  may be configured to display the average, maximum, and minimum values (called “avg/max/min”) for packet loss, jitter, and round-trip delay over a 24-hour testing session on a graphical user interface (GUI), a sample of which is shown in FIG. 3. The web server  116  may be configured to generate web-based graphical reports that indicate daily packet loss avg/max/min, daily results of the 15-minute histogram reports (e.g., reporting the percent of time in each 15-minute testing window where packet loss was less than or equal to 1%), daily jitter avg/max/min, and daily round-trip delay avg/max/min. The graphs can be generated for any pair of VoIP PoPs  104  in the VoIP network  102  using the buttons on the left side of the GUI, indicated in dashed lines  302 . The buttons on the right side of the GUI, indicated in dashed lines  304 , correspond with the 15-minute summary reports that are the basis of the graphed results. If a user wants to examine a particular 15-minute summary report in detail after viewing the graphical results, the user may select a 15-minute Summary Report button by using, for example, a keyboard or a mouse. The web server  116  can display the selected 15-minute Summary Report in the same browser window, or alternatively, generate a new browser window. When a user selects one of the buttons, indicated in dashed lines  306 , say, for example, “Max Chicago=&gt;Dallas”, the web server  116  highlights the line in the corresponding graph for the UDP stream transmitted from Chicago to Dallas during that particular 24-hour testing session.  
         [0038]    Other embodiments are within the scope of the following claims.