# Patent

## Title
Detecting interfering packet streams in packet networks

## Abstract

A method for estimating the network-layer topology of a telecommunications network is described. In particular, the illustrative embodiment of the present invention estimates the existence and connectivity of nodes in the topology based on the detection of network-wide end-to-end path intersections. This is based on the assumption that pairs of streams of packets that share a common node will interfere and that the interference can be detected in the received streams. In general, this interference is manifested as jitter. By transmitting streams on each pair of end-to-end paths in the network, and detecting interference (or a lack of interference) a matrix of path intersections for the network can be created. Using logic and supposition, the topology of the network can be estimated using the matrix of path intersections. Once the estimate of the topology is complete, the maintenance and operation of the network can proceed based on the topology.

## Background
FIELD OF THE INVENTION
The present invention relates to telecommunications in general, and, more particularly, to the maintenance and operation of telecommunications networks.
BACKGROUND OF THE INVENTION
There are many situations in the maintenance and operation of a telecommunications network when it is useful to know the logical or network-layer topology of the network. In some cases, the network-layer topology of the network is known from those who constructed it, and in some other cases the network infrastructure can provide or determine the topology. In other cases, however, the network-layer topology is not known and must be estimated based on end-to-end measurements without the support of the network infrastructure. There are techniques in the prior art that attempt to do this, but they all exhibit limitations and disadvantages. Therefore, the need exists for a new method of estimating the network-layer topology of a network using end-to-end measurements.
SUMMARY OF THE INVENTION
The present invention provides a method for estimating the network-layer topology of a telecommunications network without some of the costs and disadvantages for doing so in the prior art. For example, the illustrative embodiment of the present invention estimates the existence and connectivity of nodes in the network based on the interference of streams of packets that traverse the network. When two streams of packets traverse two end-to-end paths in the network and one stream causes jitter in the other, it can be deduced that the two paths share a common node.
In accordance with the illustrative embodiment, each pair of streams is transmitted with a particular temporal pattern. The pattern of one stream—the interference stream—is designed to induce jitter in the streams with which it shares a network node (e.g., a queue, a processor, etc.). The pattern of the interference stream is also designed to induced jitter that is distinguishable from background jitter. The pattern of the second stream—the probe stream—is designed to capture jitter caused by the interference stream in such a way that it can be readily distinguished from background jitter. By transmitting an interference stream and a probe stream on each pair of end-to-end paths in the network, all of the end-to-end path intersections for the network can be created.
Using logic and supposition, the topology of the network can be estimated using the knowledge of which end-to-end paths intersect and which do not. The topology of the network can only be estimated and not conclusively deduced because networks with different network-layer topologies can yield the same combination of end-to-end path intersections. For many applications, however, an estimated topology, even if imperfect, is useful.



BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 depicts a schematic diagram of the illustrative embodiment of the present invention.
FIG. 2 depicts a flowchart of the salient tasks associated with the operation of the illustrative embodiment of the present invention.
FIG. 3 depicts a flowchart of the salient tasks associated with the operation of task 201.
FIG. 4 depicts a flowchart of the salient tasks associated with the operation of task 304.
FIG. 5 depicts a flowchart of the salient tasks associated with the operation of task 401.
FIG. 6 depicts a flowchart of the salient tasks associated with the operation of task 402.
FIG. 7 depicts a flowchart of the salient tasks associated with the operation of task 202.
FIG. 8 depicts a flowchart of the salient tasks associated with the operation of task 203.
FIG. 9 depicts a diagram of the probe pattern, which comprises 7 seconds of small packets with spacing of 5 ms.
FIG. 10a depicts a diagram of periodic interference pattern.
FIG. 10b depicts a diagram of aperiodic interference pattern.
FIG. 11a depicts a plot of zi versus tri without interference by a periodic interference pattern.
FIG. 11b depicts a plot of zi versus tri with interference by a periodic interference pattern.



DETAILED DESCRIPTION
FIG. 1 depicts a schematic diagram of the illustrative embodiment of the present invention. The illustrative embodiment comprises: packet network 101 and six nodes 102-1 through 102-6 connected to packet network 101.
Packet network 101 comprises hardware and software, in well-known fashion, and is capable of transporting a stream of packets from any node to any other node. It will be clear to those skilled in the art how to make and use packet network 101. In accordance with the illustrative embodiment, the network-layer topology of packet network 101 is not initially known to applications on nodes 101-1 through 101-6, but is estimated using the methodology described in detail below.
Each of nodes 102-1 through 102-6 comprises hardware and software to enable it to perform the functionality described below. In accordance with the illustrative embodiment, each of nodes 102-1 through 102-6 is identical, but it will be clear to those skilled in the art, after reading this disclosure, how to make and use alternative embodiments of the present invention in which some or all of the nodes are different. Although the illustrative embodiment comprises six nodes, it will be clear to those skilled in the art, after reading this disclosure, how to make and use alternative embodiments of the present invention that comprise any number of nodes.
FIG. 2 depicts a flowchart of the salient tasks associated with the operation of the illustrative embodiment of the present invention.
In task 201, the illustrative embodiment generates a path intersection matrix A for packet network 101. The path intersection matrix A represents the knowledge of which end-to-end paths in packet network 101 intersect and which do not. To populate the path intersection matrix A, the illustrative embodiment empirically tests every pair of end-to-end paths in packet network 101 to determine whether they intersect or not. The process of testing every pair of end-to-end paths in packet network 101 to determine whether they intersect or not is described in detail below and in the accompanying figures.
In accordance with the illustrative embodiment, the following notation is used. Packet network 101 is represented as a graph G=(V,E) where V is the set of nodes (routers and hosts) and E is the set of edges representing network-layer connectivity between the nodes on V. The set N⊂V are nodes 102-1 through 102-6. The set of directional end-to-end paths between nodes in N is represented as P. A path p is considered an end-to-end path if both of its end nodes are in N. The cardinality of P is |P|==|N|×(|N|−1). An end-to-end path from niεN to njεN is represented as pi,j, where pi,jεP.
Path intersection matrix A has dimensions |P|×|P|. Each element Aa,b,c,d=1 if pa,bεP and pc,dεP intersect at the network layer, and Aa,b,c,d=0 otherwise. The populated path intersection matrix A for packet network 101 is depicted in Table 1.







TABLE 1







Path Intersection Matrix A for Packet Network 101













p1,2
p1,3
. . .
p6,4
p6,5


















p1,2
1
1
. . .
0
0



p1,3
1
1
. . .
0
0



. . .
. . .
. . .
. . .
. . .
. . .



p6,4
0
0
. . .
1
1



p6,5
0
0
. . .
0
1










Although the illustrative embodiment empirically tests every pair of end-to-end paths in packet network 101 to determine whether they intersect or not, it will be clear to those skilled in the art, after reading this disclosure, how to make and use alternative embodiments of the present invention which empirically tests fewer than every pair of end-to-end paths in packet network 101.
In task 202, the illustrative embodiment estimates the topology of packet network 101 based on the path intersection matrix A generated in task 201. The topology of packet network 101 can only be estimated and not conclusively deduced because multiple networks with different, albeit similar, topologies can yield the same path intersection matrix. For many applications, however, an estimated topology, even if imperfect, is useful. It will be clear to those skilled in the art what applications require perfect knowledge of the topology and what applications can function satisfactorily with imperfect knowledge. Task 202 is described in detail below and in the accompanying figures.
In task 203, the illustrative embodiment transmits one or more packets through packet network 101 and directs those packets to be re-directed to or away from one or more specific nodes based on the topology as estimated in task 202. As is well known to those skilled in the art, an estimate of the topology of packet network 101 is useful for many applications including, but not limited to:

    
    
        i. transmitting one or more packets to transit specific nodes and to avoid specific nodes for the purposes of fault avoidance, and
        ii. transmitting redundant one or more packets to transit alternative paths and nodes, and
        iii. transmitting one or more packets to transit nodes that are “closer” than other nodes.

Task 203 is described in detail below and in the accompanying figure.

    
    


Generating Path Intersection Matrix A—FIG. 3 depicts a flowchart of the salient tasks associated with the operation of task 201. The execution of tasks 301 through 306 generates the empirical data for one element in path intersection matrix A, and, therefore, tasks 301 through 306 are performed for each pair of paths in P.
In task 301, the illustrative embodiment transmits a stream of probe packets sa,b in a probe pattern on path pa,b FIG. 9 depicts a diagram of the probe pattern, which comprises one small packet every five milliseconds for a total of seven seconds. It will be clear to those skilled in the art, however, after reading this disclosure, how to make and use alternative embodiments of the present invention in which the probe pattern has any duration, packet size, packet number, and packet spacing.
In task 302, the illustrative embodiment transmits an interference stream of packets sc,d in an interference pattern on path pc,d at the same time that probe stream sa,b is transmitted on path pa,b. The salient desirable characteristic of the interference stream sc,d is that if it shares a network-layer node (e.g., queue, processor, etc.) with the probe stream sa,b, the interference stream should “interfere” with the probe stream by imparting a temporal pattern of jitter to the probe stream that corresponds to the interference pattern. In accordance with the illustrative embodiment, the interference pattern is characterized by a combination of a scalar k, which represents the number of bursts in the interference pattern, and a vector of k−1 inter-burst intervals.
FIG. 10a depicts a diagram of the interference pattern, which comprises 6 bursts of 1500-byte packets transmitted back-to-back at a one second inter-burst interval. It will be clear to those skilled in the art, however, after reading this disclosure, how to make and use alternative embodiments of the present invention in which the interference pattern comprises any number of bursts of any length at any inter-burst interval, whether periodic as in FIG. 10a or aperiodic as depicted in FIG. 10b. Aperiodic interference patterns are advantageous over periodic interference patterns because they can be more easily distinguished from noise than periodic patterns.
In task 303, the illustrative embodiment receives the probe stream sa,b at node nb in well-known fashion, and records for each packet i its arrival time tri.
In task 304, the illustrative embodiment detects the interference (or lack of interference) of the probe stream sa,b as received at node nb by the interference stream sc,d. This is described in detail below and in the accompanying figures.
In task 305, the illustrative embodiment deduces that paths pa,b and pc,d intersect when and only when the illustrative embodiment detects, in task 304, interference of the probe stream sa,b by the interference stream sc,d.
In task 306, the illustrative embodiment populates element Aa,b,c,d of path intersection matrix A with a 1 when the illustrative embodiment deduces that paths pa,b and pc,d intersect in task 305 and with a 0 when the illustrative embodiment deduces that paths pa,b and pc,d do not intersect.
Although the illustrative embodiment empirically tests two pairs of end-to-end paths at a time, it will be clear to those skilled in the art, after reading this disclosure, how to empirically test a plurality of pairs of end-to-end paths using multiple probe streams and a single interference stream. Furthermore, it will be clear to those skilled in the art, after reading, this disclosure, how to empirically test a plurality of end-to-end paths using one or more probe streams and multiple, distinguishable interference streams.
Detect Interference of Probe Stream sa,b by Interference Stream sc,d—FIG. 4 depicts a flowchart of the salient tasks associated with the operation of task 304.
In task 401, the illustrative embodiment generates jitter pattern Ja,b,c,d based on probe stream sa,b and interference stream sc,d. Jitter pattern Ja,b,c,d is a combination of a scalar m, which represents the number of groups found and a vector of m−1 inter-group intervals. This is described in detail below and in the accompanying figure.
In task 402, the illustrative embodiment compares jitter pattern Ja,b,c,d to the interference pattern. In accordance with the illustrative embodiment, the interference pattern is deemed to be detected when either: (1) mε{k−Δ, . . . , k, . . . , k+Δ}, wherein Δ is an integer, or (2) at least k−ψ of the k−1 inter-burst intervals temporally correspond to k−ψ of the m−1 inter-group intervals, wherein ψ is a positive integer. In accordance with the illustrative embodiment, Δ=1 and ψ=3, but it will be clear to those skilled in the art, after reading, this disclosure, how to make and use alternative embodiments of the present invention that have any value of Δ (e.g., Δ=0, 1, 2, 3, 4, 5, 10, etc.) and any value of ψ (e.g., ψ=1, 2, 3, 4, 5, 10, etc.). Task 402 is described in detail below and in the accompanying figure. It will be clear to those skilled in the art, however, after reading this disclosure, how to make and use alternative embodiments of the present invention that use another test for detecting interference of the probe stream by the interference stream.
Generate Jitter Pattern Ja,b,c,d—FIG. 5 depicts a flowchart of the salient tasks associated with the operation of task 401.
In task 501, the illustrative embodiment identifies the temporally-shifted packets in the probe stream sa,b by calculating the temporal shift zi for each packet i in the probe stream transmitted by node na at time tsi and received node nb at time tri:

zi=(tri−tr(i−1))−(tsi−ts(i−1))  (Eq. 1)

wherein ts(i−1) and tr(i−1) are the sending and receiving times of the i−1th packet, respectively. When there is no interference caused by interference stream sc,d, a plot of zi versus tri looks like that depicted in FIG. 11a, but when there is interference, a plot of zi versus tri looks something like that depicted in FIG. 11b. In FIG. 11b, the six groups of temporally-shifted packets, caused by the six periodic bursts in interference stream sc,d, respectively, are clearly visible and distinguishable from the unshifted packets. Quantitatively, however, packet i is only considered temporally shifted if it satisfies at least one of Tests 1 though 4 and either Test 5 or Test 6.

Test 1 is satisfied if:

zi<Q1−2D  (Test 1)

where Q1 is the first quartile (25th percentile) of Z and D is the difference between the 90th and 10th percentiles of Z.

Test 2 is satisfied if:

zi>Q3+2D  (Test 2)

where Q3 is the third quartile (75th percentile) of Z.

Test 3 is satisfied if:

zi<Q1−1.5IQR  (Test 3)

where IQR=Q3−Q1.

Test 4 is satisfied if:

zi>Q3+1.5IQR  (Test 4)

Test 5 is satisfied if zi is less than the 10th percentile of Z, and test 6 is satisfied if zi is greater than the 90th percentile of Z. It will be clear to those skilled in the art, after reading this disclosure, how to make and use alternative embodiments of the present invention that use other tests and combinations of tests for identifying the temporally-shifted packets.
In task 502, the illustrative embodiment groups the temporally-shifted packets that are within E seconds of one another into m groups, wherein E is a real number and m is a positive integer. In accordance with the illustrative embodiment, E=200 ms, but it will be clear to those skilled in the art, after reading this disclosure, how to make and use alternative embodiments of the present invention which use other values of E.
In task 503, the illustrative embodiment computes the m−1 inter-group intervals where the arrival time for each group equals the median arrival time for the packets in that group.
Compare fitter Pattern Ja,b,c,d to Interference Pattern—FIG. 6 depicts a flowchart of the salient tasks associated with the operation of task 402.
In task 601, the illustrative embodiment compares k (i.e., the number of bursts in the interference pattern) to m (i.e., the number of groups in jitter pattern Ja,b,c,d).
In task 602, the illustrative embodiment compares the k−1 inter-burst intervals to the m−1 inter-group intervals computed in task 401 to determine if at least k−3 inter-burst intervals temporally correspond to k−3 inter-group intervals.
Estimate Topology of Packet Network 101 Based on Path Intersection Matrix A—FIG. 7 depicts a flowchart of the salient tasks associated with the operation of task 202. The illustrative embodiment employs four deductions, two suppositions, and a rule of thumb to estimate the network-layer topology of packet network 101 based on path intersection matrix A. By considering all of these as simultaneous truths, an estimate of the topology of packet network 101 can be made using the data in path intersection matrix A. If the application of a deduction and a supposition creates an irreconcilable conflict, the deduction overrules the supposition. Absent evidence to the contrary, a supposition is considered correct.
In task 701, when path pa,b and path pc,d intersect, the illustrative embodiment deduces the existence of at least one common node ni that is in both path pa,b and path pc,d. The existence of the node might or might not have been made previously through the application of the deductions and suppositions.
In task 702, when path pa,b and path pc,d do not intersect, the illustrative embodiment deduces that path pa,b and path pc,d do not share a common node. The existence of the node might or might not have been made previously through the application of the deductions and suppositions.
In task 703, when both path pa,b and path pa,c intersect path pd,e, the illustrative embodiment supposes the existence of at least one common node ni (in addition to node na) that is in both path pa,b and path pa,c. Here too, the existence of the node might or might not have been made previously through the application of the deductions and suppositions.
In task 704, when both path pa,b and path pc,b intersect path pd,e, the illustrative embodiment supposes the existence of at least one common node ni (in addition to node nb) that is in both path pa,b and path pc,b. Here too, the existence of the node might or might not have been made previously through the application of the deductions and suppositions to other data.
In task 705, when (1) path pa,b and path pd,e intersect at a common node ni, and (2) path pa,c does not intersect path pd,e, the illustrative embodiment deduces that path pa,c does not comprise node ni. This deduction is useful to rebut a supposition made in either task 703 or 704.
In task 706, when (1) path pa,b and path pd,e intersect at a common node ni, and (2) path pc,b does not intersect path pd,e, the illustrative embodiment deduces that path pc,b does not comprise node ni. This deduction is useful to rebut a supposition made in either task 703 or 704.
Additionally, a rule of thumb applies when two paths originate or terminate at a single node. The larger the number of paths that both paths intersect (and fail to intersect), the longer the segment (i.e., edges and nodes) shared by the two paths. This can be determined by assessing the similarity between the two column vectors in the path intersection matrix that correspond to the two paths. In other words, if two paths originate or terminate at a single node, a low Hamming distance between the two column vectors for those paths suggests a long shared segment between the paths.
It will be clear to those skilled in the art, after reading this disclosure, how to estimate the topology of any network based on these deductions, suppositions, and the rule of thumb.
Utilize Deduced Topology of Packet Network 101 to Route Packets—FIG. 8 depicts a flowchart of the salient tasks associated with the operation of task 203.
In task 801, the illustrative embodiment transmits a packet or stream of packets to:

    
    
        (a) transit a first node n1 and not transit a second node n2 in some circumstances (e.g., when the jitter pattern corresponds to the temporal pattern, etc.); and
        (b) transit the second node n2 and not transit the first node n1 in some other circumstances (e.g., when the jitter pattern fails to correspond to the temporal pattern, etc.).
    
    


In task 802, the illustrative embodiment transmits a packet or stream of packets to:

    
    
        (a) transit both a first node n1 and a second node n2 in some circumstances (e.g., when the jitter pattern corresponds to the temporal pattern, etc.); and
        (b) transit neither second node n2 nor the first node n1 in some other circumstances (e.g., when the jitter pattern fails to correspond to the temporal pattern, etc.).
    
    


It is to be understood that the disclosure teaches just one example of the illustrative embodiment and that many variations of the invention can easily be devised by those skilled in the art after reading this disclosure and that the scope of the present invention is to be determined by the following claims.

## Claims

1. A method comprising:
receiving, by a first node n1 in a packet network, a first stream of packets;
generating, by the first node n1, a jitter pattern based on the first stream of packets, wherein the jitter pattern comprises one or more temporally-shifted packets, and wherein generating the jitter pattern comprises:
(1) identifying the one or more temporally-shifted packets in the first stream of packets,
(2) grouping the one or more temporally-shifted packets that are within E seconds of one another into m groups, wherein m is a positive integer and E is a real number, and
(3) computing the m−1 inter-group intervals to generate the jitter pattern;

comparing, by the first node n1, the jitter pattern to a temporal pattern; and
transmitting a packet in the packet network to transit at least one of:
(a) a second node n2 and not transit a third node n3 when the jitter pattern corresponds to the temporal pattern; and
(b) the third node n3 and not transit the second node n2 when the jitter pattern fails to correspond to the temporal pattern.

2. The method of claim 1 wherein the temporal pattern comprises k bursts at k−1 inter-burst intervals; and
wherein comparing the jitter pattern to the temporal pattern comprises comparing m to k; and
wherein k is a positive integer.

3. The method of claim 1 wherein the temporal pattern comprises k bursts at k−1 inter-burst intervals; and
wherein comparing the jitter pattern to the temporal pattern comprises comparing the m−1 inter-group intervals to the k−1 inter-burst intervals;
wherein k is a positive integer.

4. The method of claim 1 wherein the temporal pattern is periodic.

5. The method of claim 1 wherein the temporal pattern is aperiodic.

6. A method comprising:
receiving a first stream of packets from a packet network;
analyzing the first stream of packets, wherein analyzing the first stream of packets comprises:
(a) identifying one or more temporally-shifted packets in the first stream of packets,
(b) grouping the one or more temporally-shifted packets that are within E seconds of one another into m groups, and
(c) computing the m−1 inter-group intervals to generate a jitter pattern;

comparing m to an expected value of k; and
transmitting a packet in the packet network to:
(a) transit a first node n1 and not transit a second node n2 when m ε {k−Δ, . . . ,k , . . . , k−Δ}; and
(b) transit the second node n2 and not transit the first node n1 when m ∉0 {k−Δ, . . . ,k , . . . , k−Δ};

wherein k and m are positive integers, E is a real number, and Δ is an integer.

7. The method of claim 6 wherein k is the number of bursts in a temporal pattern that comprises k−1 inter-burst intervals; and
wherein the first stream of packets comprises m groups of temporally-shifted packets at m−1 inter-group intervals; and
further comprising:
comparing the k−1 inter-burst intervals to the m−1 inter-group intervals.

8. The method of claim 6 wherein the m−1 inter-group intervals are equal.

9. The method of claim 6 wherein the m−1 inter-group intervals are not all equal.

10. A method comprising:
receiving a first stream of packets from a packet network;
identifying a plurality of temporally-shifted packets in the first stream of packets by calculating a temporal shift zi for each packet i in the first stream of packets, wherein zi=(tri−tr(i−1)) −(tsi−ts(i−1)), and wherein:
(a) zi is a real number,
(b) i is a positive integer,
(c) tsi and tri are times in which each packet i is transmitted by a first node n1 and received b a second node n2 respectively,
(d) ts(i−1) and tr(i−1) are times in which each (i−1) packet is transmitted by the first node n1 and received by the second node n2, respectively;

grouping based on the temporal shift zi calculated for each packet i, the temporally-shifted packets in the first stream of packets that are within E seconds of one another into m groups;
comparing m to an expected value of k; and
transmitting a packet in the packet network to:
(a) transit a third node n3 and not transit a fourth node n4 when m ε {k−Δ, . . . ,k , . . . , k+Δ}; and
(b) transit the fourth node n4 and not transit the third node n3 when m ∉ {k−Δ, . . . ,k , . . . , k+Δ};

wherein k and m are positive integers, Δ is an integer, and E is a real number.

11. The method of claim 10 wherein k is the number of bursts in a temporal pattern that comprises k−1 inter-burst intervals; and
wherein the first stream of packets comprises m groups of temporally-shifted packets at m−1 inter-group intervals; and
further comprising:
comparing the k−1 inter-burst intervals to the m−1 inter-group intervals.

12. The method of claim 10 wherein the m−1 inter-group intervals are equal.

13. The method of claim 10 wherein the m−1 inter-group intervals are not all equal.