REAL-TIME ESTIMATION OF LONG RANGE DEPENDENT PARAMETERS

Abstract: A method and apparatus of estimating long range dependent parameters, such as Hurst parameter H and size parameter cf, of a data stream in real-time. It includes inputting blocks of data of the data stream to a real-time discrete wavelet decomposition means (200) used to generate wavelet coefficients. A sum of squares of the coefficients at each scale is maintained together with the number of elements combined in the sum within processor means, such as a PC (304). When an estimation is required, averages of the squares of the coefficients are formed, followed by the calculation of a weighted linear regression using the averages leading to a determination of estimates of the parameters.

Description: REAL-TIME ESTIMATION OF LONG RANGE DEPENDENT PARAMETERS The present invention relates to a real-time method of estimatmg parameters used to characterise a Long Range Dependent (LRD) process such as is found in data streams. The present invention more particularly relates to a method of estimating the Hurst (H) parameter and size parameter c f of a telecommunications data traffic sequence in real-time which assists in the analysis of data in teletraffic applications. The phenomenon of LRD has recently attracted strong interest in telecommunications with the discovery of self-similar and long range dependent properties in data and communications traffic of diverse types. The investigation of the impact of this on telecommunication network performance has highlighted the need for accurate and computationally effective estimation methods for LRD parameters. Long range dependence is known to be present in a wide variety of generalised data types including data traffic in high speed telecommunications networks. The presence of LRD allows one to predict trends in data traffic where there is a strong correlation between sets of the data. A common definition of LRD is the slow, power-law like decrease at large lag of the autocovariance function of a stationary stochastic time series {x t }. Equivalently, it can be defined as the power-law divergence at the origin of the spectrum of that series: f x (υ) ~ c f |υr α , |υ| -» 0 (1) or ~ C f |υ ' ' ^ lul → O where H = (1 + α)/2 is the Hurst parameter, 0.5 < H < 1, α is the "scale" parameter and c f is the "size" parameter. 2 Essentially the LRD phenomenon states that the sum of all correlations downstream from any given time instant is always appreciable, even if individually the correlations are small. Crucially, it implies that there is no possibility of defining a characteristic time-scale beyond which correlations would have essentially disappeared, as would be the case for a process whose autocorrelation function decayed exponentially - the classical assumption. Thus, one cannot find a reference unit of time over which, for instance, some property of the data could be reliably measured. Instead of a single prominent time scale, LRD is characterised by scale invariance properties governed by the parameter which describes the relationship between scales. The simplest definition of LRD involves the two parameters and c f mentioned hereinbefore, of which α is more important. As α, and therefore H, appear in the exponent of (1) it defines the existence of the LRD phenomenon and governs the characteristic scaling behavior of a LRD process as well as statistics derived from it including basic ones such as the sample mean of the series. Thus H gives a measure of the rate of decay of correlation between sets of data. It is therefore important that α is estimated well. The parameter c f plays a major role in fixing the absolute size of LRD generated effects, the general character of which is determined by H. Estimating c f is an issue of importance for quantitative analysis, but is fraught with the same statistical difficulties intrinsic to H estimation. There exist estimators of the Hurst parameter that are either time-domain based or frequency-domain based which have poor statistical performance including high bias and/or high variance. However, estimators using wavelet analysis can provide a natural, statistically and computationally efficient estimation of the Hurst parameter H, in an unbiased 3 manner. Wavelet analysis is a tool which studies the scale dependent properties of data directly via the coefficients of a joint scale-time wavelet decomposition. The wavelet decomposition takes into account the scaling behavior of a process by examination over a multitude of scales. As such, very little needs to be assumed about the underlying process. Should evidence of LRD be found, it then offers an unbiased semi-parametric estimator which can be efficiently implemented using techniques from discrete multi-resolution analysis (MRA) (see P. Abry, P. Goncalves and P. Flandrin - "Wavelets, Spectrum Estimation, 1/f Processes", Wavelets and Statistics, Lectures Note in Statistics, Vol. 105 (1995), pp. 15-30). However, such wavelet estimators only take into account the "static" analysis of a collection of data up to a particular time instant or over a time period. As such, it is an off-line process requiring a vast amount of memory to store each and every data set for the analysis thereof. If new sets of data are required to be added to the existing collection of data and thereafter analysed, all of the data sets are still required to be maintained and therefore more memory is required for the storage of the data sets. As further data sets are added, it becomes impractical to have vast arrays of memory storage devices. There is a need to be able to analyse real-time data, for example, teletraffic data so that the effect of additional data on a telecommunications network can be considered instantly without the requirement of storing vast amounts of data sets over a particular time interval. The present invention provides for a real-time method of estimating LRD parameters of on-line data streams based on wavelet analysis without the need for large memory storage and rapidly enough to handle very high data rates. The method scales naturally so that as data transfer rates become higher over time, the method will be implementable and effective in terms of speed and memory storage. According to a first aspect of the invention there is provided a method of estimating long range dependent (LRD) parameters, such as Hurst parameter H and size parameter C f ,of a data steam in real-time, said method comprising the steps of: inputting each block of data of said data stream to a real-time discrete wavelet decomposition means; extracting wavelet coefficients generated from said wavelet decomposition means for each block of input data; maintaining a sum of squares of said wavelet coefficients and maintaining the number of elements combined in said sum at each one of a number of scales; at times when an estimation is required, the method further comprising the following steps: forming averages of said squares of said wavelet coefficients at said each one of a number of scales; performing a weighted linear regression using said averages over a range of scales; and determining estimates for the LRD parameters on the basis of said linear regression. According to a second aspect of the invention, there is provided a method of estimating LRD parameters, such as H and C f , of a data stream in real-time, said method comprising the steps of: extracting packets of data from said data stream; aggregating the packets over an n th time interval to generate a data point x (n); inputting each data point x(n) to a real-time discrete wavelet decomposition means; extracting wavelet coefficients d j generated from said decomposition means for each data point x(n); maintaining a sum S j of squares d 2 j of said wavelet coefficients and maintaining a number } of data points used in said sum S j at each scale j of a number of scales; at times when an estimation is required, the method further comprising the following steps: 5 forming averages of the squares d 2 j of said wavelet coefficients from said sum S j at each scale j ; performing a linear regression using said averages over a range of scales; and determining estimates for the LRD parameters on the basis of said linear regression. The method may further include calculating confidence intervals for said estimates. Preferably, the wavelet coefficients are discarded after the step of squaring said wavelet coefficients. Preferably, prior to the performing step, a range of scales is chosen over which the linear regression is plotted. The step of choosing the range of scales may be performed manually on long time scales or automatically according to a suitable algorithm. The wavelet decomposition means may comprise a multi-resolution algorithm means or general filter bank means. The wavelet decomposition means may be implemented in software or hardware, for example, using a digital signal processing chip. The present invention also provides for apparatus for estimating LRD parameters, such as H and C f , of a data stream in real-time, said apparatus comprising: means for receiving data packets of said data stream; pre-processor means for aggregating the received data packets over time intervals to generate respective data points x(n); a real-time discrete wavelet decomposition means for receiving each data point x (n); said decomposition means generating wavelet coefficients dj for each received data point x(n); means for calculating a sum S j of squares d 2 j of said coefficients and deriving a number n; of data points used in said sum S j at each scale j of a number of scales; memory means for storing S j and n,; 6 wherein at times when an estimation of the LRD parameters is required, said means for calculating (a) forms averages of the squares d 2 j of said coefficients from said sum S j at each scale j; (b) derives a linear regression using said averages over a range of scales, and (c) determines estimates for the LRD parameters on the basis of said linear regression. A preferred embodiment of the invention will hereinafter be described with reference to the accompanying drawings wherein: Figure 1 is a block diagram showing the algorithm used to calculate the LRD parameters from a data traffic stream; Figure 2 is a schematic diagram of a wavelet decomposition means in the form of a pyramidal filter-bank implementing a discrete wavelet transform to determine the wavelet coefficients; Figure 3 is a block diagram of hardware used to implement the algorithm of Figure 1; Figure 4 is a flow diagram showing the steps involved in calculating LRD parameters using an off-line algorithm; Figure 5 is a linear regression plot of y : versus j, and Figure 6 is a log-log plot for Ethernet data. In Figure 1, there is shown a flow diagram 10 depicting a series of steps used for calculating the LRD parameters, H and c f , in real-time starting from the initial input of the data stream to be analysed. 7 At step 20, the process commences by interfacing with the physical media carrying ethernet traffic in order to extract the traffic packets. The traffic packets are then filtered at step 30 to extract relevant data. The process then progresses to step 40 where the relevant data is aggregated over the n time interval to generate the next data point x(n) of the series to be analysed. Each data point x(n) generated is input to a real-time filter bank at step 50 which is used to extract and update the Discrete Wavelet Transform (DWT) coefficients, to be described with reference to Figure 2. After the coefficients have been determined for each data point x(n), then at step 60, each new data point is added to the existing number of data points for each scale j so that (n,+l) replaces n, as the new updated total of data points. The sum of squares S, is also updated to include the new coefficients squared at that scale. Information on the existing coefficient is discarded. Finally at step 70, at suitable time intervals, a calculation of time average μ j for each j is made and the LRD parameters H and c f are calculated to be discussed later. A discrete wavelet transform (DWT) is performed on each data point x(n) wherein a number of wavelet coefficients are produced through a wavelet decomposition process using the real-time filter bank. The wavelet transform can be understood as a more flexible form of a Fourier Transform, where the original signal is transformed, not into a frequency domain, but into a time-scale wavelet domain. The sinusoidal functions of Fourier theory are replaced by wavelet basis functions generated by simple translations and dilations of the the mother wavelet ψ 0 , itself defined via multiresolution theory (see I. Daubechies, "Ten Lectures on Wavelets", SIAM (1992)). The wavelet transform can be thought of as a method of simultaneously observing the signal at a full range of different scales or resolutions j. The wavelet coefficients of each data point x(n) essentially comprise details d, and approximations a } for each scale j. In Figure 2, there is shown a wavelet decomposition means 200 which may be a multi-resolution algorithm in the form of a real-time filter bank. Each data point 8 x(n) in the series to be analysed over an interval of time is input to the wavelet decomposition means 200 and fed through band-pass filter (BPF) 202 and downsampler 204. The wavelet coefficient, or detail, of the data point is output at the first scale as d]. Part of the input data point x(n) is fed to low-pass filter (LPF) 206, through downsampler 208 and then is split again. One portion is input to BPF 210 and downsampler 212 to extract the coefficient d 2 at scale 2. Another portion is input to LPF 214 and downsampler 216. The process is repeated to extract the coefficients up to scale j. Therefore, output di is updated for every second new data point x(n), output d 2 is updated for every fourth new x(n) and output d j is updated for every (2 J ) th new x(n). Each BPF and LPF in the wavelet decomposition means 200 is of a finite length K, so that only K input values are held in the memory of each BPF or LPF. Once a value, for example x(n) itself which is input to BPF 202, has propagated through the filter, the value is dropped or discarded. Storage requirements are therefore fixed for each filter with K. log (n) values being stored in the filter bank overall. Each of the filters may be implemented by a Finite Impulse Response (FLR) filter. The wavelet decomposition means may be implemented in software or in hardware, for example, on a DSP chip. Referring to Figure 1, after the DWT coefficients d, are updated, using x(n), in the real-time filter bank in step 50, a number of actions occur in step 60. For each x(n) that is updated at each scale j: the number of wavelet coefficients at that scale is incremented by one, that is, n, is replaced by (n,+l); the existing sum S j of squared coefficients is replaced by (S_, + d j 2 ) to produce an updated sum of squares. The coefficient value d, is then discarded as it is no longer required. Therefore, all that is required to be stored in memory is the updated sum S j and the number of data points n,. Note that it is only when a d, is updated in step 50, that it 9 is forwarded onto the next step 60 for the calculation of the new n, and S j and then d, is discarded thereafter. The new values n } , S j are stored in memory residing in the RAM. In step 70, at various time intervals suitable to the user, a time average μ j is calculated for each j, where Aj = Σ d (2) n } using the stored S j and n, for each scale. Then, based on the values for (n,) and (μ,), an estimation of the LRD parameters H and c f is performed using a wavelet-based joint estimator or off-line algorithm, to be described hereinafter. Figure 3 shows the hardware, in block form, used to implement the algorithm of Figure 1. A Network Interface Card (NIC) 302 is used to interface with, and capture traffic packets from the Ethernet 300. The Ethernet may be 10 Base T or 10 Base 2. It is then presented to PC 304 which comprises an Intel Pentium PC rurming a version of the UNLX(R) operating system named Free BSD. The PC 304 runs the traffic analysis software, written in C and performs the necessary calculations for obtaining a sum S j of squares d 2 j , deriving the n- of data points used in the sum S } . Such software may also be used to form averages of d j at each scale j, subsequently derive a linear regression, determine an estimate for H and c f and calculate any updates for S, and n, on arrival of new data points x(n) Alternatively, hardware such as DSP chips may be used to calculate a sum S j of squares d 2 derive n, and form averages μ j of d 2 } at each scale j.. The PC 304 includes a software preprocessor 306, a DWT and Estimator Unit 308 and memory unit 310. The preprocessor 306 preprocesses the traffic measurements through the use of the Berkeley Packet Filters, and outputs a time series of the number of bytes per time interval. The preprocessed traffic measurements are passed to unit 308 which 10 updates the on-going wavelet decomposition, summary statistics and Hurst parameter estimation, and outputs the results periodically to a printer, plotter or other device. The memory unit 310 is in the form of RAM and is used to store the preprocessed traffic measurements, wavelet coefficients derived from the measurements and summary statistics. Note that only the N most recent measurements and coefficients at each scale are required, where N is the length of the FIR filters used to implement the LPFs and BPFs used in the wavelet decomposition. The remainder of the data has been condensed and stored in the summary statistics. In Figure 4, there is shown a flow diagram 400 of the steps involved in estimating LRD parameters H and c f using the wavelet-based joint estimator described in "A wavelet-based Joint Estimator of the Parameters of Long-Range Dependence, Technical Report SERC-0043, 1997 by D. Veitch and P. Abry". The steps 410 to 470 are self-explanatory and what follows is a detailed description of the steps. The Wavelet Estimator In the analysis of the LRD phenomenon, the following two features, (F1,F2), of the family of wavelet basis functions play key roles: FI: The family of wavelet basis functions generated from the mother wavelet ψo are constructed from the dilation or change of scale operator: Ψ l0 (t) = j i/2 ψ 0 (7 i t) (3) This means that the analyzing family exhibits, by construction, a scale invariance feature. The LRD phenomenon can be understood as the absence of any characteristic frequency (and therefore scale) in the range of frequencies close to the origin. The LRD property can thus be interpreted as a scale invariance characteristic which is efficiently analysed by wavelets. 11 F2: ψo has a number N of zero or vanishing moments which can be freely chosen provided N > 1. By definition this means that I t k ψ 0 (t) dt ≡ 0, k = 0,...., N - 1 (but not for k > N), or equivalently that the Fourier Transform of ψo satisfies |Ψ 0 (υ)| = O (|υ| N ) at the origin. This property can be used to control divergences arising with processes having power-law spectra at the origin. For a process with a power-law spectrum such as a LRD process, these features engender the following key properties of the wavelet coefficients d j over a range of scales 2 J , j = J 1 -J 2 where the power-law scaling holds. PI: Due to FI, the scale invariance (the power-law behavior) is captured exactly: IEd j 2 = 2 jα c f C (4) where C = J |υr|Ψ 0 (υ)| 2 dυ. (5) This exact recovery of a power-law is not a trivial effect and results directly from the dilation operator underlying the design of the wavelet basis. Time-frequency or periodogram based estimates would not exhibit such a feature. P2: Due to FI and F2, the d j are a collection of random variables which are quasi-decorrelated (see "Wavelet Analysis and Synthesis of Fractional Bfownian Motion" by P. Flandrin, IEEE Trans, on Info. Theory IT-38 (1992) pp. 910-917). In particular, the long-range dependence present in the time domain representation is completely absent in the wavelet coefficient plane {j,k}. 12 Elaborating on Property P2 it has been shown that correlations in the time-scale plane decay at least hyperbolically in all directions with exponents controlled by the number of vanishing moments and corresponding to short range dependence. Since by definition the octave j = log 2 (scale), this implies exponential decay in octave j. From hereon log 2 will denote the logarithm base 2, whereas In will denote natural logarithms. The intuitive basis of the estimator can be found by analysing equation (4). Rewriting it as log 2 (IEdJj,.) 2 ) = jα + log 2 (C f C) strongly suggests a linear regression approach for estimating (α,C f ), where clearly the slope of the regression would estimate α and the intercept would be related to c f . The idea of using a log- log plot is common to many contexts when an exponent is the object of interest. The real issue is to what extent the promise of this simple linear form is realised in the resulting estimator, once the inevitable complications are taken into account. The first, essential, complication is of course that IEd , a second order quantity that can be related to the spectrum of x, is not known but must be estimated. In the present context this is the principal difficulty as it is well known (see "Statistics for Long-Memory Processes", J. Beran, Chapman & Hall (1994)) that the estimation of second order (and other) quantities in a long range dependent context is a delicate task. Here, however, property P2, the quasi-decorrelation of the d j allows us to effectively use the simple "time average" μy-Z d (6) n. where n, is the number of coefficients at octave j available to be analysed. This quantity is an unbiased and consistent estimator of IEd j 2 . (Note that μ j is the 13 sample variance of d,, since from F2 the expectation of d j is identically zero for eachj). Thus the sample data d j for n } samples at scale j of the wavelet decomposition is squared and then summed and divided by n, to form the "time average". The second complication is the non-linearity introduced by the log 2 , which biases the estimation. We will see below how this problem also can be circumvented under reasonable hypotheses. Simplifying things slightly, we confirm that the fundamental approach underlying our estimator is indeed a linear regression of log 2 (μ j ) on log 2 (2 J ) = j. A weighted linear regression will be used as the variances of the log (μ,) vary with j. Linear regression The fundamental hypothesis of linear regression is LEy, = bx, + a. We define the quantities S = Σ 1/σ S x = Σ x σ, and S ∞ = ∑x } " /σ i where σ j is an arbitrary Λ weight associated with y,. The usual unbiased estimator (b,a) of (b, a) is Λ ∑ y (S xj - Sj/ σ) _ b = = ∑ Wj y } , (8) xx ~ jύ x „ ∑ y. Sxx - S XjXσ) _ a = _ ≡ ∑ v j y j , (9) where the weights w, and υ } satisfy ∑w j = ∑ υ } = 0, ∑j w, = ∑u, = 1. Note that these conditions imply that there are always both positive and negative υ } and w,. If in addition the y s are mutually independent then the covariance matrix is given by 14 Var(b) = ∑ σ ) w y (10) U H ~ ύ x Var(z) = ∑ σ 2 2 _ v , - ■ (H) jr " J j Cov( , b) = ∑ σ ) Wj Vj = ~ Sτ , , (12) xx " x r = - Sχ /JsSχχ, (13) where r is the correlation coefficient. If X j > 0 for each j it is easy to see that r will be negative, and large in magnitude if Xj is large, as a small change in the slope "to the right" will result in an amplified change of opposite sign in the intercept. Finally, if we set σ, 2 = Var(y,), then (b Λ , a) is the minimum variance unbiased estimator (-VTVTJE) (see "Fundamentals of Statistical Signal Processing" by S.M. Kay, Prentice-Hall (1993)) with covariance matrix as above. Note that in the event of small errors in the values of the σ, 2 and small correlations between the y,, the estimator remains unbiased and its covariance matrix can be accurately estimated by the expressions just given. Thus far we have indicated that log 2 (μ,) is the variable y, of the desired linear regression satisfying IEy- j = bj + a. Since IE log 2 (μ,) ≠ log 2 (IEμ,) = jα + log 2 (c f C) in general, this cannot be exactly true, although under the conditions H1-H3 below, and also assuming n, large, it can be established that 15 l g 2 (μ ) - N ja - log, c, C, nln 2 2 where signifies equality in distribution and N(μ,σ " ) is a Gaussian random variable. In a LRD context however, the large scales are usually the most important to consider, and it is precisely there that the η are not large. Here the condition j is removed by examining the distribution of log 2 (μ j ) in more detail. It turns out that this refinement leads to only a small improvement in the estimation of α, but has very important implications for the estimation of c . Throughout the analysis it is instructive to bear in mind that the number of available detail coefficients n j essentially decreases by half as the scale is doubled, that is xi j+ j a n j /2, and therefore that n j+ ι « n2 "J where n is the length of the initial data. We assume that the following supplementary hypotheses hold true. HI : The process x, and hence the processes d(j,.), are Gaussian. H2: For fixed j the process d(j,.) is iid. H3: The processes d(j,.) and d(j',.), j ≠ j', are independent. Hypothesis HI is justified by numerical evidence which shows that the method is very insensitive to the form of the marginal distributions of x. Hypotheses H2 and H3 are both well justified by property P2 (they are separated to make it clearer which properties are needed where). These extra conditions, whilst appearing very restrictive at first glance, are in fact very reasonable in practical terms, as bome out in simulations. The reason for this is that the underlying effectiveness of the method is based on PI and P2, H1-H3 being added only to extend the quantitative analysis. Let the density of a Chi-squared variate X u d X u 2 be denoted by 16 -\ t -sa /.(*) = The mean and variance of such a variate are υ v/ I\v/2) and 2υ respectively. Also set z, = 2 jσ C f C. From HI and H2 and equations 4 and 6 we have (14) ' n n > where IEμ j = z,- as μ j is unbiased, and therefore log 2 (μ ) - log 2 Z j - log 2 n + log 2 x nj ja + log 2 c f C - log 2 π y - lnX π /ln2. (15) Thus the study of log 2 (μ j ) reduces to that of the logarithm of a Chi-squared variable. Using the relations J fO " X '1 e ' ^xx dz = — . . v r (υ)[ψ(u) - lnμ], Rεμ > μ ' 0,Reυ >0 (see Table of Integrals, Series and Products", LS. Gradshteyn and I.M. Ryzhik, Academic Press, corrected and enlarged edition (1980)), equation j 4.352), and T o x"' e^fl xf dx = —T(v) [(ψ(v) - ]nμ f ÷ C(2.v)], Reμ > Reυ > 0, (above A reference equation 2, t * 4.358) where ψ(z) = I " "(z) fT(z) is the Psi function and ζ (z,υ) is a generalised Riemann Zeta function, it is straightforward to show from the definition of f υ (x) above that 17 IE In χ υ = ψ(υ/2) + In 2, (16) Var(lnχ υ ) = ζ(2,υ/2). (17) It follows that IE \og (μ j ) = ja + \og 2 c / C + g J , (18) Var(logXμ j )) = ζ(2,n J /2)/\n 2 2, (19) where the term gj = ψ( nj /2)/ϊa2 - log « y /2). (20) a negative function of n, only, can be easily calculated for all values of n,. The term g, is a small corrective term which is substracted to account for the distorting non-linearity introduced by the log. For future reference, below are the asymptotic form for n, large of the quantities above: sy- tiy j In- 2, (21) Var(log 2 ( Mj )) ~ (22) rij ln -' 7 18 Defining the variable y } as y, = log 2 (μ,) - g j (23) we see that from the above description it is clear that under HI and H2 they obey IEy^ j a + \ g 2 C f C (24) Var(y j ) = ζ(2, nj /2)/\X2, (25) and thus satisfy the requirements for a weighted linear regression. A weighted regression estimation (b, a) of y j on j = x, is performed according to equations 8 and 9 with σ, 2 = Var(y,). An example of the regression fit using a simulated data set is given in Figure 5 where y, = log 2 (μ-)-g, is plotted against j and showing 95% confidence intervals. The 95% confidence intervals for each y J5 shown as vertical lines at each octave j, are seen to increase with j. This can be understood from equation (22), remembering that n, « n2 "J meaning that the number of data points halves with every increase in j by one. The intervals are derived from the known sample variances σ, of the estimates y, under gaussian assumptions. An LRD process is apparent between scales 4 and 10 whereas strong SRD or short range dependence is apparent for scales less than 4, and particularly for the range j=l to j=3. The vertical bars at each octave give 95% confidence intervals for the y,. The series is simulated farima (0,d,2) with d=0.25 (α=0.50) and Ψ = [-2, -1] implying c f = 6.38. Selecting (jι > J 2 ) = (4,10) identifies the relevant scaling range allowing an accurate estimation despite the strong SRD: a = 0.53 ± 0.7, c f = 6.0 with 4.5 < c f < 7.8. In Figure 6, there is shown a log-log plot for real Ethernet data. The data is from an Ethernet trace and contains in excess of 30 million observations which took 19 only a few seconds to analyse. It plots log 2 (μ j ) versus j. It shoes an example of the wavelet based scale analysis for continuous time Ethernet data. The asymptotic LRD behaviour is seen to enter at octave j=14. The estimate of the self-similarity parameter H=l - β \ 2 is H=0.8. By using the wavelet decomposition means, the static LRD estimation method provides significant advantages in terms of memory storage, memory usage and calculation times. Using it, the input data can be split into blocks, analysed and recombined, so that memory problems are not encountered in treating data of arbitrary length n. The run time complexity of the method is very low, of the order n or O(n), making it very suitable for the analysis of very large data sets. These advantages make the method suitable for real-time implementation where the data is collected and the LRD parameters estimated on a continuous, on-the-fly basis. In the future, when the total amount of data to be processed rises from megabits to gigabits, and even terabits, storage problems will not arise as the requirements of the real-time algorithm vary as the logarithm of the data length. The real-time version described here and the working preferred implementation prove that these potential real-time advantages can be achieved in practice.

Claims: 20 CLAIMS 1. A method of estimating long range dependent (LRD) parameters, such as Hurst parameter H and size parameter C f ,of a data steam in real-time, said method comprising the steps of: inputting each block of data of said data stream to a real-time discrete wavelet decomposition means; extracting wavelet coefficients generated from said wavelet decomposition means for each block of input data; maintaining a sum of squares of said wavelet coefficients and maintaining the number of elements combined in said sum at each one of a number of scales; at times when an estimation is required, the method further comprising the following steps: forming averages of said squares of said wavelet coefficients at said each one of a number of scales; performing a weighted linear regression using said averages over a range of scales; and determining estimates for the LRD parameters on the basis of said linear regression. 2. A method according to claim 1 wherein, for each updated scale, the existing sum of squares of said wavelet coefficients is replaced by a new sum of squares equivalent to the existing sum added to the square of each new wavelet coefficient. 3. A method according to claim 1 or claim 2 wherein, for each updated scale, the existing number of elements combined in said existing sum is incremented by the number of new elements at each scale. 4. A method according to claim 2 or claim 3 wherein the value of each coefficient is not retained or stored after each update. 5. A method according to any one of the previous claims wherein each block of data input to the decomposition means is not retained or stored. 6. A method according to any one of the previous claims wherein the step of forming averages includes forming a time average at each scale equivalent to the 21 updated sum of squared coefficients divided by the updated number of elements combined in the updated sum. 7. A method according to any one of the previous claims wherein the linear regression is plotted and prior to the performing step, a range of scales is chosen. 8. A method according to any one of the previous claims further comprising the step of calculating confidence intervals for each of the derived estimates. 9. A method of estimating LRD parameters, such as H and C f , of a data stream in real-time, said method comprising the steps of: extracting packets of data from said data stream; aggregating the packets over an n th time interval to generate a data point x (n); inputting each data point x(n) to a real-time discrete wavelet decomposition means; extracting wavelet coefficients d j generated from said decomposition means for each data point x(n); maintaining a sum S j of squares d 2 , of said wavelet coefficients and maintaining a number n, of data points used in said sum S j at each scale j of a number of scales; at times when an estimation is required, the method further comprising the following steps: forming averages of the squares d 2 j of said wavelet coefficients from said sum S j at each scale j; performing a linear regression using said averages over a range of scales; and determining estimates for the LRD parameters on the basis of said linear regression. 10. A method according to claim 9 wherein, for each updated scale j , the sum S j is replaced by a new sum (S j +d^ ) to yield an updated sum of squares. 11. A method according to claim 9 or claim 10 wherein, for each x(n) updated at scale j, the number of data points n, , or equivalently the number of coefficients at scale j, is incremented by one such that n, is replaced by (n,+l). 12. A method according to any one of claims 9 to 11 wherein the value of each d j is not retained or stored after each update. 22 13. A method according to any one of claims 9 to 12 wherein each data point x (n) input to the decomposition means is not retained or stored. 14. A method according to any one of claims 9 to 13 wherein the step of forming averages comprises forming a time average μ j at each scale j such that 2) _ Sj Vj = Σ d nj J ) using the maintained S j and n j for each scale. 15. A method according to claim 14 further comprising the step of calculating a random variable y j where y j = log 2 (μ j ) - g j where g j is a small corrective term. 16. A method according to claim 15 further comprising the step of plotting y j against j with confidence intervals about j based on σ j where σ j is an arbitrary weight associated with j . 17. A method according to claim 16 wherein after the plotting step a scaling range is chosen on which to base the linear regression. 18. A method according to claim 17 wherein following selection of the scaling range, a weighted linear regression of y j on j is performed in the selected scaling range with weight σ j 2 . 19 A method according to claim 18 wherein an estimate of H (or a where a = 2H-1) is obtained from the slope of the regression and an estimate of Cf is obtained from the intercept of the regression. 20. A method according to claim 19 further comprising the step of calculating confidence intervals of the estimate of LRD parameters H and C f . 21. Apparatus for estimating LRD parameters, such as H and C f , of a data stream in real-time, said apparatus comprising: 23 means for receiving data packets of said data stream; pre-processor means for aggregating the received data packets over time intervals to generate respective data points x(n); a real-time discrete wavelet decomposition means for receiving each data point x (n); said decomposition means generating wavelet coefficients d j for each received data point x(n); means for calculating a sum S s of squares d 2 j of said coefficients and deriving a number n, of data points used in said sum S j at each scale j of a number of scales; memory means for storing S j and n,; wherein at times when an estimation of the LRD parameters is required, said means for calculating (a) forms averages of the squares d 2 j of said coefficients from said sum S j at each scale j; (b) derives a linear regression using said averages over a range of scales, and (c) determines estimates for the LRD parameters on the basis of said linear regression. 22. Apparatus according to claim 21 wherein, for each updated j, the sum S j , is replaced by a new sum (S j + d, 2 ), to yield an updated sum of squares, which new sum is subsequently stored in said memory means. 23. Apparatus according to claim 21 or claim 22 wherein, for each x(n) updated at scale j, the number of data points n, , or equivalently the number of coefficients at scale j, is incremented by one , such that n, is replaced by (n, +1) and the updated value (n,+ 1 ) is subsequently stored in said memory means. 24. Apparatus according to any one of claims 21 to 23 wherein said means for calculating is implemented in software. 25. Apparatus according to any one of claims 21 to 23 wherein said means for calculating calculates the sum S,, derives n, and forms said averages using hardware and derives said linear regression and said estimates using software. 26. Apparatus according to any one of claims 21 to 25 wherein the value of each d, is not retained or stored in said memory means. 27. Apparatus according to any one of claims 21 to 26 wherein each data point x(n) received by said decomposition means is not subsequently retained or stored by said memory means. 24 28. Apparatus according to any one of claims 21 to 27 wherein said pre-processing means, said decomposition means and said memory means are part of a computing processor such as a PC. 29. Apparatus according to any one of claims 21 to 28 wherein said decomposition means comprises a multi-resolution algorithm means. 30. Apparatus according to claim 29 wherein said multi-resolution algorithm means is a real-time filter bank comprising a series of low-pass filters (LPFs) and band-pass filters (BPFs). 31. Apparatus according to claim 30 wherein said LPFs and said BPFs are finite impulse response filters. 32. A method or system substantially as hereinbefore described with reference to the accompanying drawings.

Inventors: ABRY PATRICE, ROUGHAN MATTHEW, VEITCH DARRYL NEIL, ABRY, PATRICE, ROUGHAN, MATTHEW, VEITCH, DARRYL, NEIL
Applicants: ABRY PATRICE, CENTRE NAT RECH SCIENT, ERICSSON AU PTY LTD, MELBOURNE INST TECH, ROUGHAN MATTHEW, VEITCH DARRYL NEIL, ABRY, PATRICE, CENTRE NATIONAL DE RECHERCHE SCIENTIFIQUE (CNRS), ERICSSON AUSTRALIA PTY. LTD., ROUGHAN, MATTHEW, ROYAL MELBOURNE INSTITUTE OF TECHNOLOGY, VEITCH, DARRYL, NEIL
Application year: 1999
Main classification: H04L12
IPC codes: H04L12