Patent ID: 12192030

DETAILED DESCRIPTION

FIG.1illustrates system100adapted according to embodiments configured to optimize the throughput of bonded multiple variable bandwidth connections by adjusting a tunnel bandwidth weighting schema during a data transfer session. System100includes multiple sites102and104, which each comprise at least one communications router106and108. Communication routers106and108may be embodied as multi-WAN routers which support aggregating the bandwidth of multiple Internet connections. Communications routers106and108are connected over network110. Network110may comprise a local area network (LAN), metropolitan area network (MAN), wide area network (WAN), wireless network, the public switched telephone network (PSTN), the Internet, an intranet, an extranet, etc.FIG.1illustrates system100adapted according to embodiments configured to optimize the throughput of bonded multiple variable bandwidth connections by adjusting a tunnel bandwidth weighting schema during a data transfer session. System100includes multiple sites102and104, which each comprise at least one communications router106and108. Communication routers106and108may be embodied as multi-WAN routers which support aggregating the bandwidth of multiple Internet connections. Communications routers106and108are connected over network110. Network110may comprise a local area network (LAN), metropolitan area network (MAN), wide area network (WAN), wireless network, the public switched telephone network (PSTN), the Internet, an intranet, an extranet, etc.

Site102and router106may comprise M connections112, and site104and router108may comprise N connections114. Connections112and114are data connections for communicating information within network110between sites102and104. In the illustrated embodiment, M is equal to 3 and N is equal to 2; however, these values may vary according to desired routers and configurations. Connections112and114may have similar or differing bandwidth capabilities. Further, connections112and114may comprise different types of WAN connections, such as a WiFi, cable, DSL, T1, 3G, 4G, satellite connections, and the like. It is also noted that site102and site104may be thought of as both a sender or receiver, and discussions regarding the functionality of either site may be implemented on the other site. In other words, system100may be implemented as a symmetrical network. There is no limitation that M and N are as the same as the number of physical network interfaces of routers106and108respectively. For example, router106has five physical network interfaces for connecting to different WANs and M may be one, ten, one hundred or one thousand. The M×N tunnels may be aggregated to form a bonded tunnel, a bonded VPN or a bonded connection. Tunnels and connections are used interchangeably in the present invention to describe logical connections between two network interfaces, which may be physical or virtual network interfaces.

FIG.2illustrates the change of latency of different tunnels, which are established between a first network node and a second network node, through a period of time. As shown in the table, there are six tunnels, Tunnel A to Tunnel F, aggregated to form a bonded connection. Encapsulating packets encapsulate data packets, which are received by one network node and destined to the other network node or destinations reachable through the other network node. At t=0 s, among Tunnels A-F, Tunnel D has the lowest latency (5 ms) while Tunnel F has the highest latency (100 ms), and the first network node sent a plurality of encapsulating packets to the second network node. It is expected that encapsulating packets sent through Tunnel D will arrive at the second network node the earliest while encapsulating packets sent through Tunnel F will arrive last. However, there are myriad factors affecting the latency of each tunnel, resulting an ever-changing latency of a tunnel. At t=10 s, among Tunnels A-F, Tunnel A has the highest latency (80 ms) while Tunnel B has the lowest latency (5 ms). Accordingly, when a plurality of data packets of a session are sent through the bonded connection in sequence, it is likely that the order at the receiving end is different from that at the sending end due to the varying latency of different tunnels. This causes undesirable jitter among data packets at the receiving end. One of the objectives of present invention is to reduce this undesirable jitter.

To monitor the bandwidth of the various tunnels116, some embodiments of the present invention encapsulate each transmitted IP packet with various information. Transmitted IP packet may be called as encapsulating packet in the present invention as an IP packet is encapsulated in the transmitted packet.FIG.3, a prior art, illustrates an example embodiment showing the type of information300which may be encapsulated in a transmitted IP packet. Version field302may contain information about the protocol version being utilized and protocol type field303may contain the protocol type of the payload packet. Tunnel ID field304may contain an identifier to identify the current tunnel of the IP packet. Advanced Encryption Standard (AES) initialization vector field306may be a 32-bit field and may contain an initialization vector for AES encryption. Global sequence number field308may contain a sequence number which is utilized to re-sequence each of the packets for various sessions into the proper order when they have emerged from their respective tunnels. Per tunnel sequence number field310may represent a sequence number that is assigned to each packet routed to a particular tunnel.

The per tunnel sequence number discussed is able to be used for packet drop monitoring and re-ordering packet, not cannot be used for reducing jitter for data belong to a session.

FIG.4A, also a prior art, illustrates an example embodiment of the type of information400which may be encapsulated in a feedback packet which is sent to the transmitting router in order to report packet drop rates or other bandwidth related data received at the receiving end router. Type field402may include data regarding the type of data that will be included in data 1 field404and data 2 field406. Data 1 field404and data 2 field406may contain any information which may be used to assist the router in determining tunnel information with regard to the number of tunnels, bandwidth of tunnels, number of dropped packets in a tunnel, and the like. An example of possible values of the type field402in the data fields404and406is shown in the chart ofFIG.4B, a prior-art.

As disclosed in prior art, the information which is encapsulated in transmitted IP packets, such as shown inFIGS.3-4may also be used for packet buffering and re-sequencing. Because each tunnel's latency can be different, when two consecutive packets of the same TCP session are sent to a VPN peer over a bonded VPN tunnel, they may not arrive in sequence because they are routed via two different tunnels. If the TCP session receives the out-of-sequence packets from the VPN, the TCP session will slow down due to TCP retransmissions. Accordingly, the receiving end should buffer the packets that come too early until either the slower packets arrive, or an expiration time has passed. With such buffering, late packets that come prior to an expiration time will be forwarded to the destination device in sequence. This buffering assists in the optimization of end-to-end throughput.

It is noted that embodiments described herein are, at times, discussed in the context of a VPN connection. These discussions are presented in order to show an example embodiment of a bonded connection. The inventive concepts described in claimed herein are not limited to such connections. In fact, any connection where sufficient data may be obtained and exchanged in order to dynamically monitor the bandwidth of a plurality of communication paths which are being used in a data transfer session may be implemented with the embodiments of the present invention.

Embodiments, or portions thereof, may be embodied in program or code segments operable upon a processor-based system (e.g., computer/processing system or computing platform) for performing functions and operations as described herein. The program or code segments making up the various embodiments may be stored in a computer-readable medium, which may comprise any suitable medium for temporarily or permanently storing such code. Examples of the computer-readable medium include such tangible computer-readable media as an electronic memory circuit, a semiconductor memory device, random access memory (RAM), read only memory (ROM), erasable ROM (EROM), flash memory, a magnetic storage device (e.g., floppy diskette), optical storage device (e.g., compact disk (CD), digital versatile disk (DVD), etc.), a hard disk, and the like.

Embodiments, or portions thereof, may be embodied in a computer data signal, which may be in any suitable form for communication over a transmission medium such that it is readable for execution by a functional device (e.g., processor) for performing the operations described herein. The computer data signal may include any binary digital electronic signal that can propagate over a transmission medium such as electronic network channels, optical fibers, air, electromagnetic media, radio frequency (RP) links, and the like, and thus the data signal may be in the form of an electrical signal, optical signal, radio frequency or other wireless communication signal, etc. The code segments may, in certain embodiments, be downloaded via computer networks such as the Internet, an intranet, LAN, MAN, WAN, the PSTN, a satellite communication system, a cable transmission system, and/or the like.

FIGS.6A-6Dillustrate time sequences of data packets and transmitted IP packets received and transmitted by communications router106and108according to embodiments of the present invention.

In one example, for illustration purpose, in time sequence611, communications router106received data packets602a-602dat time t=1 ms, t=4 ms, t=7 ms and t=8 ms respectively from a sending device. Data packets602a-602d, for illustration purpose, belong to the same session and are destined to a destination device reachable through communications router108. For readability purpose, contents of data packets602a-602dare labelled with “Data-1”, “Data-2”, “Data-3” and “Data-4” respectively. As communications router106transmits transmitted IP packets to communications router108, communications router106is the sending router while communications router108is the receiving router.

In time sequence612, transmitted IP packets603a-603dare transmitted by communications router106to communications router108through at least one connection. Transmitted IP packets603a-603dencapsulate data packets602a-602drespectively and are transmitted at time t=11 ms, t=14 ms, t=17 ms and t=18 ms respectively. Shaded areas of transmitted IP packets603a-603dillustrate information which may be created by communications router106and may be encapsulated in transmitted IP packets603a-603d. The details of the information is illustrated inFIG.7.

There is no limitation that all transmitted IP packets must be sent through the same WAN interface or the same tunnel. There is also no limitation that all data packets must be distributed in any particular order or weights among WAN interfaces or tunnels. For example, transmitted IP packets603a,603b,603cand603dmay be transmitted through a first tunnel, a second tunnel, the first tunnel and a third tunnel respectively.

FIG.7illustrates an example embodiment showing the type of information700which may be encapsulated in a transmitted IP packet. Comparing toFIG.3, fields711-713are added. Field711is to record the arrival time of a data packet, such as one of data packets602, at the sending router. The arrival time may be used by receiving router, such as communications router108, to determine when to forward data packets to destination device. In one variant, field711is to record the transmission time of a transmitted IP packet. Before the transmitted IP packet is transmitted by the sending router, field711is updated with a time. Therefore, the receiving router may rely on field711to forward consecutive data packets with corresponding time difference between the consecutive data packets. It is preferred to have the time recorded by the sending router at least in microsecond precision for network that is not faster than 100 Gbps. If the recorded time is not precise enough, there will be many transmitted IP packets having the same time value in field711. Receiving router may then not be able to precisely forward consecutive data packets with precise time difference. For network that is faster than 100 Gbps, the time record may be in nanosecond precision.

The time value may be based on an internal clock of the sending router. However, there is no limitation that the time value in field711must be a correct clock time. There is also no limitation on the size of field711. The larger the size field711is, the less frequent the time value needs to be reset or start over.

Session identification field712is used for holding a value for a session identification, which is used to identify a session. There are myriad ways to identify a session. For example, a session may be identified by deep packet inspection (DPI) or by a combination of source address, source port number, destination address and destination port number. For example, the value for a session identification may be a hash value based on the combination of source address, source port number, destination address and destination port number. In another example, the value for a session identification may be created based on the arrival time of the first data packet of a session. Subsequent transmitted IP packets encapsulating data packets of the session will use the same value for the session identification.

Session sequence number field713is for holding a value for a session sequence number. The session sequence number is to allow data packets to be re-ordered at communications router108, which is the receiving router. For example, consecutive data packets belonging to the same session may have consecutive session sequence number.

In one variant, global sequence number field308may be omitted when session identification field712and session sequence number713are used, as session sequence number may be used for ordering data packets belonging to the same session and/or transmitted IP packets belonging to the same session.

There is no limitation that ordering of fields302-312and711-713must be in the order illustrated inFIG.7. There is also no limitation that length of a field must be the same as the number of bits illustrated inFIG.7.

It is appreciated that if the sending router could correctly record the arrival time or transmission time of each data packets, the receiving router may then be able to determine the correct sequence of the data packets. In one variant, when field711is used for recording arrival or transmission time, session sequence number field713may be omitted as receiving router may rely on the arrival time or transmission time to determine the sequence of forwarding data packets to the destination device. In one variant, global sequence number field308and per tunnel sequence number field310may be omitted when field711is used for recording arrival or transmission time. In one variant, global sequence number field308and per tunnel sequence number field310may be omitted when session sequence number field713and session identification field712are used to determine the correct sequence of transmitted IP packets.

Referring back to time sequence612, the session sequence numbers encapsulated in information of transmitted IP packets603a-603dare one, two, three and four respectively in order to indicate that transmitted IP packet603ais encapsulating the first data packet of the session; transmitted IP packet603bis encapsulating the second data packet of the session; and so forth.

It is preferred that the time differences between consecutive data packets602received and the time differences between consecutive transmitted IP packets603transmitted are the same. For example, the time difference between data packets602aand602b(t=1 ms and t=4 ms) is preferred to be about the time difference between transmitted IP packets603aand603b(t=11 ms and t=14 ms). This may reduce jitter when transmitted IP packets arrived at communications router108. In another example, data packets602dis received right after data packets602c. In order to reduce jitter, transmitted IP packet603dis preferred to be transmitted right after data packet603c. Information encapsulated in transmitted IP packets603a-603dmay include arrival time of data packets602a-602drespectively. Alternatively, the information may include transmission time of transmitted IP packets603a-603drespectively.

There is no limitation that transmitted IP packets603must be transmitted in the same sequence as data packets602are received or must be transmitted in the same connection. For example, transmitted IP packets603aand603care transmitted through a first connection, such as connections112-1, while transmitted IP packets603band603dare transmitted through a second connection, such as connections112-3.

In time sequence613, for illustration purpose, transmitted IP packets603a-603dare received by communications router108at time t=31 ms, t=34 ms, t=36 ms and t=39 ms respectively. Transmitted IP packets are not received in the correct sequence. The time differences between consecutive transmitted IP packets603a-603dare also changed and result in jitters.

In order to forward data packets602a-602din the same sequence and reduce jitter, transmitted IP packets603a-603dare first buffered in one or more queues when they arrived at communications router108. Data packets602a-602dare then retrieved by decapsulating transmitted IP packets603a-603drespectively. Finally, data packets602a-602dare forwarded to destination device in the same sequence and with substantially the same time difference from respective prior data packets in order to reduce jitter. The time difference may be determined using respective time value stored in field711of the transmitted IP packets.

There is no limitation that transmitted IP packets of different sessions must be stored in different queues. The number of queues can be less than the number of sessions, therefore, transmitted IP packets of different sessions could be stored in the same queue. The receiving router could identify the session of each transmitted IP packet according to the session identification in field712. When determining the expiration time of each transmitted IP packets, even though there may be transmitted IP packets of different sessions stored in the same queue, the calculation is based on one or more transmitted IP packets with same session identification.

FIG.8illustrates a high-level flow diagram of a receiving router depicting a method800for reducing jitter when forwarding data packets to a destination device. For illustration purpose, the receiving router is communications router108andFIG.8is discussed in conjunction withFIGS.6A-6D. It should be appreciated that the particular functionality, the order of the functionality, etc. provided inFIG.8is intended to be exemplary of operation in accordance with the concepts of the present invention. Accordingly, the concepts herein may be implemented in various ways differing from that of the illustrated embodiment.

The clock at the receiving router should be at least in microsecond precision and in nanosecond precision for network that is not faster than 100 Gbps and for network that is faster than 100 Gbps respectively. The clock may be a BIOS clock of the receiving router or based on BIOS clock. When a transmitted IP packet is received by the receiving router, a processing unit will store the transmitted IP packet in queue along with an expiration time of the transmitted IP packet. The expiration time may be stored in the same queue, in a list, in a table or any non-transitory computer readable storage medium at the receiving router. When the expiration time of a transmitted IP packet is reached, the data packet encapsulated in the transmitted IP packet will then be forwarded to the destination device. For transmitted IP packet603b, it is received by receiving router at time t=31 ms at step801, and then stored by a processing unit of the receiving router into a queue corresponding to the session identification in step802. For illustration purpose, transmitted IP packet603bis the first transmitted IP packet received in time sequence613. The session identification is encapsulated in the information, which is illustrated by the shaded area of the transmitted IP packet603b.

As session sequence number of transmitted IP packet603bis two, processing unit is able to determine that there is a transmitted IP packet with session sequence number one has not arrived yet at time t=31 ms, and as a result, processor of receiving router may not process transmitted IP packet603band/or will not forward the data packet encapsulated in IP packet603bto the destination device of the data packet. Expiration time of the transmitted IP packet603bmay be set to any value at this point. The expiration time, for example, is set to t=44 ms at step802. Transmitted IP packet603bwill wait and be kept stored in the queue at step804. In one variant, the expiration time is set at 61 ms at step802for a longer buffer so as to improve sequence order. In one variant, the expiration time is set at 36 ms at step802for a shorter buffer so as to forward the data packet earlier.

By time t=44 ms, the time of the transmitted IP packet603bis or upon being expired in step803. The transmitted IP packet603bis dequeued at step805. Then data packet602bis decapsulated from transmitted IP packet603b. Finally, data packet602bis transmitted to the destination device at t=44 ms at step806.

For transmitted IP packet603a, it encapsulates data packet602a, which was received by sending communication device earlier than data packet602b. When transmitted IP packet603ais received by receiving router at step801, it is then stored in the queue corresponding to the session identification at step802. As the session sequence number of transmitted IP packet603ais prior to session sequence number of transmitted IP packet603b, transmitted IP packet603ais placed at a position earlier than the position of transmitted IP packet603bin the queue at step802. Further, as indicated by the session sequence number, the processing unit of the receiving router is able to determine that data packet602ais the first packet in the session. In another example, no data packet will be forwarded to the destination device until a transmitted IP packet with session sequence number one arrived. This ensures that the first data packet forwarded to the destination device is the data packet with session sequence number one and may reduce jitter.

A buffer period is assigned to transmitted IP packet603a. The buffer period of transmitted IP packet603ais based on a time value configured by an administrator of receiving router or retrieved from a remote management server. The benefits of having a large buffer period including: reducing possibility of forwarding out of sequence data packets to the destination device; and reducing jitter when forwarding data packets to the destination device. The benefits of having a short buffer period including: smaller size queue is required and forwarding data packets earlier.

In one embodiment, the value of buffer period is configurable by a user or an administrator through command line, a web page or a graphical user interface (GUI) of the communication router. The value of buffer period may also be pre-configured by the vendor of the communications router. In one example, same value of buffer period is applied to all transmitted IP packets. In another example, different sessions or flows may have different buffer period values. Therefore, a GUI may allow a user or an administrator to enter buffer period values corresponding to different sessions or flows. For example, buffer period value for a session of source IP address A, source port A, destination IP address B and destination port B may be configured by a user to be 10 ms while buffer period value for a session of source IP address C, source port C, destination IP address D and destination port D may be configured to be 300 ms.

For example, if the buffer period is configured to be 7 ms for the first data packet in a session, the transmitted IP packets encapsulating the first data packet of a session will be stored in the queue for 7 ms and then the first data packet will be forwarded to the destination device. Therefore, transmitted IP packet603awill be stored in the queue for 7 ms. Subsequent data packets of the data session will be forwarded to the destination device after being stored in the queue shorter or longer than 7 ms, depending on the time variation experienced by each transmitted IP packet during transmission.

In one variant, transmitted IP packets that do not encapsulate the first data packet in the session may also be stored in the queue according to a second buffer period, such as at least one half of the buffer period or up to twice of the buffer period. This allows flexibility in determining expiration time of a transmitted IP packet. Alternatively, a user or administrator of the receiving router may input the range of buffer period to the receiving router.

In one variant, transmitted IP packets that do not encapsulate the first data packet in the session will be stored in the queue for the buffer period if the queue is empty.

By reference to the expiration time of the transmitted IP packets encapsulating the first data packet, the expiration time of the other transmitted IP packets encapsulating the subsequent data packets could be calculated. Further, storage time of each transmitted IP packets may also be determined by subtracting its expiration time from the time it arrived the receiving router. Storage time of a transmitted IP packet may change according to the upcoming received transmitted IP packet. In general, a transmitted IP packet having an earlier session sequence number should have an earlier expiration time in the queue. If a transmitted IP packet having an earlier session sequence number arrives late, it is possible that adjustment on the storage time of the currently stored transmitted IP packet may be required. It should be noted when storage time of a currently stored transmitted IP packet is adjusted, expiration time of the currently stored transmitted IP packet is also adjusted. In such case, the storage time of the currently stored transmitted IP packet may be prolonged. If a transmitted IP packet having a later session sequence number arrives early, adjustment on the storage time of the currently stored transmitted IP packet may be required, and the storage time of the currently stored transmitted IP packet may be shortened.

As the expiration time of the transmitted IP packets603ais t=41 ms, refer to time sequence611, the time difference between data packets602aand602bat the sending router is 3 ms (4 ms−1 ms). Therefore, as shown in time sequence614, the expiration time of the transmitted IP packets603bis t=44 ms (41 ms+3 ms). Further, the storage time of the transmitted IP packets603bis calculated to be 13 ms (44 ms−31 ms).

For transmitted IP packet603c, refer to time sequence611, the time difference between data packets602aand602cat the sending router is 6 ms (7 ms−1 ms). Therefore, as shown in time sequence614, the expiration time of the transmitted IP packets603cat the receiving router is t=47 ms (41 ms+6 ms). Further, the storage time of the transmitted IP packets603cis calculated to be 11 ms (47 ms−36 ms).

As the transmitted IP packets may arrive the receiving router in random order, some of the transmitted IP packets with prior sequence number may arrive late. In one variant, if the calculated expiration time of a newly received transmitted IP packet is already expired. Therefore, the data packet encapsulated in the transmitted IP packet should be decapsulated and forwarded to the destination device without further delay. In this case, storage time of the newly received transmitted IP packet should be zero, or the newly received transmitted IP packet will not be stored in the queue. Other transmitted IP packets received but not yet dequeued will remain in the queue, the expiration time of other currently stored transmitted IP packet may remain the same. In one variant, if the calculated expiration time of a newly received transmitted IP packet is already expired. The newly received transmitted IP packet will still be stored in the queue but dequeued immediately. In one variant, if the calculated expiration time of a newly received transmitted IP packet is already expired, expiration time of the currently stored transmitted IP packets will be adjusted. A new expiration time is calculated for currently stored transmitted IP packets so as to restore the sequence; and to reduce jitter by keeping the time difference of expiration time among the transmitted IP packets close to the time difference of the time value in field711among the transmitted IP packets.

There is no limitation that when determining expiration time, arrival time of data packets at the sending router must be used. In one example, the transmission time of transmitted IP packet at the sending router may be used. For instance, refer to time sequence612, the time difference between transmitted IP packets603aand603cat the sending router is 6 ms (17 ms−11 ms). The expiration time for transmitted IP packet603cis calculated to be 47 ms (41 ms+6 ms). A processing unit of the receiving router may use the time value in field711of the information of the transmitted IP packet as one of a plurality of parameters to determine the expiration time.

At step803, the processing unit of the receiving router will determine whether the expiration time of the transmitted IP packets is already expired. When the expiration time of a transmitted IP packet is or upon being expired, step805will then be performed to dequeue the transmitted IP packet and decapsulate the transmitted IP packet to retrieve the data packet(s) encapsulated. At step806, the data packet(s) will then be transmitted to the destination device. Using transmitted IP packet603bas an example, data packet602bwill be transmitted to the destination device at step806when its corresponding expiration time is or upon being expired and after steps803and805are performed.

FIG.9illustrates a high-level flow diagram of a receiving router depicting a method900for reducing jitter when forwarding data packets to a destination device. In this embodiment, based on method800, step802is replaced by steps901-904.

In the circumstances that a long delay is experienced, for a period of time, during transmission of transmitted IP packets of a session. As a result, it is possible that all the priorly arrived transmitted IP packets of a session had already been dequeued, decapsulated and forwarded to the destination device, resulting an empty queue within the same session. When the queue is empty, there is no currently stored transmitted IP packet. When transmitted IP packets of the same session arrive at the receiving router, its calculated expiration time maybe already expired, therefore, the transmitted IP packets will be decapsulated and forwarded to the destination device immediately so as to avoid further delay. However, the sequence correctness and the time differences among the remaining packets are neglected. Method900provides an extra procedure to reduce the jitter caused under this situation.

In step901, after receiving a transmitted IP packet, the processing unit of the receiving router108will determine the status of the queue in step901. If the queue is not empty, step902will be performed accordingly. The expiration time is determined based on time differences between the time value in field711of the transmitted IP packet and the time value in field711of one or more currently stored transmitted IP packets. If the queue is empty, there is no currently stored transmitted IP packet, there is no reference to determine the expiration time. Therefore, a buffer period is used to determine the expiration time of the transmitted IP packet in step903. In one variant, the expiration time and session sequence number of the last transmitted IP packet dequeued are recorded and used for reference to determine the expiration time.

The buffer period may be predefined by the manufacturer of receiving router108, or may be set by a user or an administrator through command line, a web page or a GUI of the communication router. In one variant, the expiration time is the arrival time of the transmitted IP packet at the receiving router plus the buffer period. In step904, the transmitted IP packet is stored in the queue for dequeuing.

FIG.10illustrates a high-level flow diagram of a receiving router depicting a method1000for reducing jitter when forwarding data packets to a destination device. At step1001, a receiving router, such as communication router108, receives a transmitted IP packet. At step1002, a processing unit will determine whether the transmitted IP packet has a sequence number earlier than all sequence numbers of transmitted IP packets currently stored in the queue.

If the transmitted IP packet has an earliest sequence number, it is likely that the transmitted IP packet arrived late. In order to reduce jitter, expiration time of transmitted IP packets currently stored in the queue will be increased to reflect the delay caused by the transmitted IP packet at step1003. Therefore, the forwarding of data packets encapsulated in transmitted IP packets currently stored will be delayed.

If the transmitted IP packet does not have an earliest sequence number, the transmitted IP packet may arrive early or may arrive late (but not too late to affect jitter). Therefore step1003can be skipped.

At step1004, the transmitted IP packet is stored in the queue corresponding to the session identification stored in session identification field712. In one variant, if transmitted IP packet is already expired, step1004is skipped, and the data packet encapsulated in the transmitted IP packet will then be forwarded.

FIG.11illustrates a high-level flow diagram of a receiving router depicting a method1100for reducing jitter when forwarding data packets to a destination device.FIG.11is discussed in conjunction withFIGS.6A-6D. In this embodiment, each transmitted IP packet received by the receiving router is only allowed to be stored in the queue for a maximum storage time (MST). There are two stages to determine the expiration time for each transmitted IP packets.

At step1101, the receiving router, such as communications router108, received a transmitted IP packet through a plurality of connections from a sending router, such as communications router106. According to the session identification, the processing unit is able to identity the session and its corresponding queue. At step1102, a processing unit will perform the first stage, which is to calculate preliminary expiration time of the transmitted IP packet received.

The preliminary expiration time is calculated based on the time difference with one or more currently stored transmitted IP packets in the queue. For example, for illustration purpose, expiration time for transmitted IP packet603ais 41 ms. As the time difference between data packets602dand602ais 7 ms, the preliminary expiration time of transmitted IP packet602dis then set to t=48 ms (41 ms+7 ms).

At step1103, the processing unit will determine whether the preliminary storage time is larger than the MST. If time is now at 40 ms and the MST is configured to be 10 ms, then transmitted IP packet603dwill be stored in the queue for 8 ms (48 ms−40 ms), which is less than the MST (8 ms<10 ms). Therefore, step1106is performed, the storage time of transmitted IP packet603dis set to the value of the preliminary storage time, which is 8 ms. Therefore, the transmitted IP packet603dwill be stored in the queue until t=48 ms (40 ms+8 ms). Accordingly, when expiration time of the transmitted IP packet is or upon being expired, the transmitted IP packet will be dequeued from the queue and be decapsulated to retrieve the data packet encapsulated in the payload of the transmitted IP packet, and then the data packet will be forwarded to the destination device.

At step1103, if the storage time according to the preliminary expiration time is larger than the MST, the processing unit will perform the second stage which is illustrated by steps1104and1105.

At step1104, the expiration time of the transmitted IP packet received is set to the MST. For example, for illustration purpose, expiration time for transmitted IP packet603ais 41 ms. As the time difference between data packets602cand602ais 6 ms, the preliminary expiration time of transmitted IP packet603cis then set to t=47 ms (41 ms+6 ms). If the time is now at 36 ms and the MST is configured to be 5 ms, then transmitted IP packet603cwill be stored in the queue for 11 ms (47 ms−36 ms), which is larger than the MST (11 ms>5 ms). Therefore, the storage time of transmitted IP packet603cis set to the value of the MST, which is 5 ms. Transmitted IP packet603cis stored in the queue until t=41 ms (36 ms+5 ms).

At step1105, expiration times of transmitted IP packets currently stored in the queue will be adjusted in order to reduce jitter. This also ensures that no transmitted IP packet received will be stored in the queue longer than the MST.

After adjustment at step1105, if the expiration time of a currently stored transmitted IP packet becomes expired, that transmitted IP packet will be decapsulated immediately. The corresponding data packet will be forwarded to destination device accordingly.

In one variant, if there is no transmitted IP packet in the queue, the preliminary storage time of the transmitted IP packet is set to the MST, or a value substantially close to the MST.

In another example illustrated byFIGS.12A-12Bviewing in conjunction with the high-level flow diagram illustrated inFIG.11, communications router106received data packets1202a-1202cin time sequence1201at time t=1 ms, t=2 ms, and t=8 ms respectively from a sending device. Transmitted IP packets1203a-1203cencapsulate data packets1202a-1202crespectively and arrived at communication router108in time sequence1211at t=9 ms, t=11 ms and t=10 ms respectively.

For illustration purpose, the MST is configured to be 16 ms. The storage time for the first arrived transmitted IP packet of a session is set to be the MST. When transmitted IP packet1203aarrived at t=9 ms, it will be stored in the queue for 16 ms and therefore, data packet1202ais scheduled to be forwarded to the destination device at t=25 ms (9 ms+16 ms). When transmitted IP packet1203carrived at t=10 ms, as the time difference between data packet1202cand1202ais 7 ms, then at step1102, the preliminary expiration time is set to t=32 ms (25 ms+7 ms). The storage time is based on the preliminary expiration time, therefore, will be 22 ms (32 ms−10 ms). As the storage time is larger than the MST (22 ms>16 ms) in step1103, the storage time is set to the MST at step1104. As a result, expiration time for transmitted IP packet1203cis then set to be t=26 ms (10 ms+16 ms). Expiration time of transmitted IP packet1203a, which is currently stored, will then be adjusted to t=19 ms (26 ms−7 ms) at step1105.

When transmitted IP packet1203barrives at t=11 ms, the preliminary expiration time will be calculated to be t=20 ms as the expiration time of transmitted IP packet1203ais adjusted to t=19 ms and data packet1202bis received by communication router1061 ms after data packet1202aand 6 ms before data packet1202c. The storage time based on the preliminary expiration time will then be 9 ms and less than the MST at step1103. The expiration time will then be set to t=20 ms at step1106.

In one variant, the MST is still configured to be 16 ms but the storage time for the first arrived transmitted IP packet of a session is only one half of the MST. For readability, the preliminary percentage of the MST for the first transmitted IP packet to be stored is referred as PPMST. PPMST, in this variant, is set to 50%. When transmitted IP packet1203aarrived at t=9 ms, it will be stored in the queue for 8 ms (16 ms×50%) and therefore, data packet1202ais scheduled to be forwarded to the destination device at t=17 ms (9 ms+8 ms). When transmitted IP packet1203carrived at t=10 ms, the preliminary expiration time is set to t=24 ms (17 ms+7 ms). The storage time according to the preliminary expiration time, therefore, will be 14 ms (24 ms−10 ms). As the storage time is smaller than the MST in step1103, the expiration time is then set to the same value of the preliminary expiration time, which is t=24 ms, at step1106.

When transmitted IP packet1203barrives at t=11 ms, the preliminary expiration time will be calculated to be t=18 ms (17 ms+1 ms) as the expiration time of transmitted IP packet1203ais t=17 ms and data packet1202bis received by communication router1061 ms after data packet1202aand 6 ms before data packet1202c. The storage time based on the preliminary expiration time will then be 6 ms and less than the MST at step1103. The expiration time will then be set to t=18 ms at step1106.

There is no limit on the value of MST. There is also no limit on the PPMST as long as it is not more than one hundred percent. As illustrated in the last illustration, when the first arrived transmitted IP packet of a session is set to a value less than the MST, it reduces the probability of adjusting the expiration time for currently stored transmitted IP packets. One of the observed benefits is to reduce the effect on jitter for transmitted IP packets that arrive much earlier than other transmitted IP packets.

In one embodiment, the value of the MST is configurable by a user or an administrator through command line, a web page or a graphical user interface of the communication router. The value of the MST may also be retrieved from a remote server or pre-configured by the vendor of the communications router. In one example, one MST is applied for all transmitted IP packets. In another example, different sessions or flows may have different MST values. Therefore, a GUI may allow a user or an administrator to enter the MST values corresponding to different sessions or flows. For example, the MST value for a session of source IP address A, source port A, destination IP address B and destination port B may be configured by a user to be 10 ms while the MST value for a session of source IP address C, source port C, destination IP address D and destination port D may be configured by a user or an administrator to be 300 ms.

There is no limitation that there is only one MST for the receiving router. In one variant, one MST is applicable to all connections that are bonded between the transmitted router and receiving router. Therefore, all connections or tunnels in a bonded site-to-site VPN connections on multi-WAN routers have the same MST. In one variant, there is one MST for each connection of a bonded connection. In one variant, there is one MST for each session of each connection of a bonded connection.

In one embodiment, the value of the PPMST is also configurable by a user or an administrator through command line, a web page or a GUI of the communication router. The value of the PPMST may also be retrieved from a remote server or pre-configured by the vendor of the communications router. In one example, one MST is applied for all transmitted IP packets. In another example, different sessions or flows may have different PPMST values. Therefore, a GUI may allow a user or an administrator to enter the PPMST values corresponding to different sessions or flows. For example, the PPMST value for a first session is 10% while the PPMST value for a second session is 80%.

There is no limitation that there is only one PPMST for the receiving router. In one variant, one PPMST is applicable to all connections that are bonded between the transmitted router and receiving router. In one variant, each connection of a bonded connection has its own PPMST. In one variant, each session has its own PPMST. In one variant, each session and each connection of a bonded connection has its own PPMST.

In one variant, when there are no transmitted IP packets in the queue, the expiration time of the transmitted IP packet is set to the MST, or a value substantially close to one half of the MST. This may help reducing jitter in the scenario when the next transmitted IP packet arrived late and/or a future transmitted IP packet arrive much earlier than other transmitted IP packets.

There is no limitation that embodiments of present inventions are limited to Internet Protocol packets. These embodiments are also applicable to other non-circuit-switching communication protocols, including Ethernet.

FIG.5illustrates an exemplary processor-based system500which may be employed to implement the systems, devices, and methods according to certain embodiments. Central processing unit (CPU)501is coupled to system bus502. CPU501may be any general purpose CPU, or may be a special purpose CPU designed to implement the above teachings. The present disclosure is not restricted by the architecture of CPU501(or other components of exemplary system500) as long as CPU501(and other components of system500) supports the inventive operations as described herein. CPU501may execute the various logical instructions described herein. For example, CPU501may execute machine-level instructions. When executing instructions, CPU501becomes a special-purpose processor of a special purpose computing platform configured specifically to operate according to the various embodiments of the teachings described herein.

System500also includes random access memory (RAM)503, which may be SRAM, DRAM, SDRAM, or the like. System500includes read-only memory (ROM)504which may be PROM, EPROM, EEPROM, or the like. RAM503and ROM504hold user and system data and programs, as are well known in the art.

System500also includes input/output (I/O) adapter505, communications adapter511, user interface adapter508, and display adapter509. I/O adapter505, user interface adapter508, and/or communications adapter511may, in certain embodiments, enable a user to interact with system500in order to input information.

I/O adapter505connects storage device(s)506, such as one or more of hard drive, compact disc (CD) drive, floppy disk drive, tape drive, etc., to system500. The storage devices are utilized in addition to RAM503for the memory requirements associated performing the operations discussed in the above embodiments. Communications adapter511is adapted to couple system500to network512, which may enable information to be input to and/or output from system500via such network512(e.g., the Internet or other wide-area network, a local-area network, a public or private switched telephony network, a wireless network, any combination of the foregoing). User interface adapter508couples user input devices, such as keyboard513, pointing device507, and microphone514and/or output devices, such as speaker(s)515to system500. Display adapter509is driven by CPU501to control the display on display device510. Display adapter509transmits instructions for transforming or manipulating the state of the various numbers of pixels used by display device510to visually present the desired information to a user. Such instructions include instructions for changing state from on to off, setting a particular color, intensity, duration, or the like. Each such instruction makes up the rendering instructions that control how and what is displayed on display device510. User interface adapter508and display adapter509are optional. For example, a router may not need user interface adapter508and display adapter509as the router can be configured remotely through network512.

It shall be appreciated that the present disclosure is not limited to the architecture of system500. For example, any suitable processor-based device may be utilized for implementing the above teachings, including without limitation routers, personal computers, laptop computers, computer workstations, multi-processor servers, and even mobile telephones. Moreover, certain embodiments may be implemented on application specific integrated circuits (ASICs), or very large scale integrated (VLSI) circuits. In fact, persons of ordinary skill in the art may utilize any number of suitable structures capable of executing logical operations according to the embodiments.

Although embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims. Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification. As one of ordinary skill in the art will readily appreciate from the disclosure of the present invention, processes, machines, manufacture, compositions of matter, means, methods, or steps, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein may be utilized according to the present invention. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.