# Statistical Methods within Network Neutrality Monitoring Tools

Nada Dandachi, Patrick Maillé, Bruno Tuffin

► To cite this version:

Nada Dandachi, Patrick Maillé, Bruno Tuffin. Statistical Methods within Network Neutrality Monitoring Tools. 2025. hal-05228725

HAL Id: hal-05228725

https://inria.hal.science/hal-05228725v1

Preprint submitted on 28 Aug 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Statistical Methods within Network Neutrality
Monitoring Tools

Nada Dandachi

Inria, Univ. Rennes, CNRS, IRISA

Rennes, France

nada.dandachi@inria.fr

Patrick Maillé

IMT Atlantique, IRISA, UMR CNRS 6074

Rennes, France

patrick.maille@imt.fr

Bruno Tuffin

Inria, Univ. Rennes, CNRS, IRISA

Rennes, France

bruno.tuffin@inria.fr

August 28, 2025

Abstract

Network neutrality is a sensitive topic worldwide. Laws are passed and
regulators are expected to monitor providers activities to check if they
conform to the enacted rules. Therefore, tools need to be designed towards
that goal but require strong theoretical validations if used for justifying
legal actions. Our goal is to review the statistical grounds of the proposed
tools, focusing in particular on Wehe, co-designed by the French regulator
ARCEP. We provide a statistical analysis of the method used in Wehe,
compared with standard ones.

# 1 Introduction

*Network neutrality* has been the topic of raging debate for at least the past
20 years in the telecommunications domain [16]. Different but rather close
definitions exist, a known one being that adopted in 2014 by the European
parliament<fnref n="1" />:

<footnote n="1">¹See https://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP//NONSGML+AMD+A7-2014-0190+237-244+DOC+PDF+VO//EN</footnote>
The principle of “net neutrality” means that traffic should be treated equally, without discrimination, restriction or interference, independent of the sender, receiver, type, content, device, service or application.

Network neutrality has become a key concept of the so-called *open Internet* but has been the subject of many interrogations. It has for example been repealed in the USA in 2017, over concerns (among other things) that it was against innovation. There is also a concern about the possibility to circumvent the packet-oriented regulation in the increasingly complex Internet ecosystem, through other actors that could behave in a non-neutral way, such as content delivery networks and all other intermediates between end users and content. For more on the debate, the reader is advised to look at [7, 12, 16, 22, 25].

Whether network neutrality is adopted or not, it is of primary importance to develop tools monitoring the behavior of ISPs, to investigate whether they are manipulating traffic: it would help to understand when such actions are leading to limitations of the use of the network, and help the regulators to ensure that ISPs conform to the law, potentially leading to sanctions. Several tools have been deployed and presented in the literature, but surprisingly not so many, in particular if we list the maintained ones; see [4, 9] for surveys and Section 2 below.

With monitoring tools ultimately allowing to potentially sue offenders, that is to serve for legal purposes, the methods used have to be i) documented and detailed, and ii) based on strong theoretical grounds to avoid complaints of abusive accusations.

For this reason, it is worthwhile investigating the justifications of existing tools, discuss their theoretical validity, mostly based on statistical methods. Most tools focus on the technical side (e.g., measuring transmission delays or estimating throughput) but are limited in terms of the description of the statistical methods, probably because it is not their focal point. There are in general in research papers brief mentions of the involved mathematics without further analysis. Though, it is of primary importance when the measurements and their analysis can lead to accusations of non-neutral behaviors. Our paper aims at pointing out this issue and at discussing the validity of the tools.

At least, any used heuristics needs to be conservative in the sense that it should avoid so-called *false positives* (i.e., wrong accusations of misbehavior), even if at the same time it reduces the detection of real non-neutrality (true positives).

We will take a particular consideration to a specific tool, namely Wehe [18, 21], since co-developed by a regulator and officially used in this context. We will describe the statistical technologies, make tests and propose areas of improvement. Wehe basically compares the throughputs of a presumed discriminated flow with a similar one designed to be non-discriminated. It provides an answer to a potential non-neutrality question, but to our knowledge no extensive analysis of the considered method has been performed, nor a tuning of the parameters and comparison with standard methods. That is the main purpose of this paper.

Section 2 describes the decision methodology used in Wehe and also lists
the ones of other existing tools. Section 3 highlights why further study is of interest, to assess the accuracy of tests based on heuristics and the potential limitations of Wehe. Section 4 validates the tool, by performing comparisons with “classical” statistical tests. Finally Section 5 concludes and gives directions for future research.

# 2 Decision methodologies in Wehe and other tools

Several neutrality monitoring tools exist but to our knowledge, only a specific one is advertised by a regulator: Wehe, that has been co-designed by the French regulator ARCEP. Since our goal is to discuss the validity and justification of the tool if used for legal purposes, we focus on the non-neutrality detection methodology and its theoretical grounds, an important axis. We will then briefly describe how decisions are made in other developed tools.

## 2.1 Wehe

To detect potential violations of net neutrality laws, Wehe, developed by Northeastern University and publicized by the French regulator ARCEP, aims at identifying traffic differentiation across various applications in the mobile environment. It is an open-source [20] testing tool that analyzes the traffic generated by an application to determine whether an operator might be throttling or prioritizing some data traffic.

### 2.1.1 Wehe behavior

Wehe works by simulating using a service on an Internet Service Provider's (ISP) network, to see how that ISP treats actual traffic generated by that online service. As of May 2025, the tool tests traffic from 29 applications that are likely to be subject to differentiation: 18 video streaming apps, 4 music and 7 conferencing ones. Typical examples are Netflix, Spotify, Microsoft Teams and Youtube.

When differentiation occurs, differentiation policies are applied through traffic-shaping middleboxes, that rely on keyword-based deep packet inspection (DPI) to identify traffic from the targeted applications.

To operate, Wehe first captures and records some traffic generated by the application, e.g., by streaming a video on Netflix. This recorded trace is then embedded into the Wehe app. During a test run, the app retrieves the relevant trace, enabling the Wehe client and Wehe server to replay the original interaction between them. The same trace will then be replayed each time the specific application will be tested. The traces are recorded and replayed in a manner that preserves inter-packet timings for both TCP and UDP packets. This is done in order to make them nearly identical to the original traffic: the client and server thus re-enact the back and forth between them by relying on the recordings of the replays. As the traffic they exchange goes through the ISP, in the case of differentiation it is controlled and filtered through the traffic shapers in place.
In practice, Wehe performs two replays of the network traffic generated by the target application. The first replay, referred to as the original replay, is performed with no modifications whatsoever, exposing the traffic to any policies adopted by the middleboxes. For the second replay, the payloads of all packets are just random bits (without changing the packets' sizes), making the service unidentifiable by the middleboxes [13]; it is therefore referred to as the randomized replay.

Throughout the replays, the tool captures the average received throughput on the client side every 0.22 seconds. Then, after running both replays, Wehe relies on a complex decision process in order to compare the throughput distributions of the two replays and finally provide a positive or negative response regarding potential service differentiation.

## 2.1.2 Wehe differentiation assertion process

To assert whether there is differentiation or not, Wehe uses three statistical tools.

* First, a two-sample Kolmogorov-Smirnov (KS) test is performed. It is a non-parametric test that is used to examine whether two independent samples have been drawn from the same distribution.

Consider two samples of i.i.d random variables $(X_i)_{1\le i\le n}$ and $(Y_j)_{1\le j\le m}$, and their respective empirical cumulative distribution functions (CDFs) $F_n(x) = \frac{1}{n}\sum_{i=1}^n 1(X_i < x)$ and $F_m(x) = \frac{1}{m}\sum_{i=1}^m 1(Y_i < x)$ where $1(\cdot)$ stands for the indicator function.

Under the null hypothesis that the two samples in consideration come from the same distribution, KS examines the maximum vertical distance $d$ between their respective CDFs:

$$d = \max_x(|F_n(x) - F_m(x)|).$$

The null hypothesis is then rejected with a confidence level $\alpha$ if the test statistic $d$ is larger than a tabulated threshold, or accepted otherwise [6].

In the context of network differentiation, we aim to detect if the original and the randomized replays have been drawn from the same distribution. $(X_n)_{n\ge 1}$ and $(Y_m)_{m\ge 1}$ will be the throughputs measured over successive intervals of time 0.22 ms for respectively the un-randomized and randomized replays.

* In addition to the standard KS test, Wehe incorporates the **jackknife resampling** technique as a supplementary validation step.

Originally used in NetPolice [28] (also discussed in Section 2.2), this method was later adapted into Wehe's framework. As illustrated in Figure 1, it involves repeatedly drawing random subsets—specifically, half of the samples from each of the two distributions—and applying the KS test to these subsets. This resampling step is performed $r$ times, and if at least $\beta$% of the resulting tests yield the same conclusion as the original KS test
on the full dataset, the result of the original KS test is considered valid. Jackknife is claimed to improve the confidence interval coverage accuracy and to reduce bias in sets with a limited number of samples.

<figure><img src="image_2.png" /><figcaption>Figure 1: Schematic of jackknife resampling</figcaption></figure>

* The third statistical component of the tests is the “**area test**”, which was proposed by Wehe as a way to counter the sensitivity of the KS test. In fact, Wehe reasons that the KS test is sensitive to small differences in performance due to the fact that it only considers the difference in distributions along one axis (the y-axis). Therefore, instead of looking at the maximal vertical distance between the CDFs, wehe computes the area *a* between the CDFs of both throughput distributions. This area is then normalized by the minimum of the peak throughputs observed in each distribution. Final differentiation is detected if two conditions are met: the KS test indicates a statistically significant difference between the distributions, *and* the normalized area exceeds a predefined threshold *t*. In the application, *t* is set as 0.1, chosen from a large set of experiments and believed to be representative of a threshold for a “relevant” accuracy for small sample sizes *n* and *m*.

Figure 2 illustrates the differences between the KS and the area tests.

Figure 3 summarizes the decision-making process used by Wehe. The process begins with data collection, followed by a sequential application of statistical tests discussed above. First, the area test is applied, and its result is recorded. Next, the KS test is performed on the full dataset with a 95% confidence level.
<figure><img src="image_3.png" /><figcaption>Figure 2: Statistical tests in Wehe [18]</figcaption></figure>

To validate the outcome of the KS test, the jackknife resampling method is then used. The interpretation of the results leads to one of the following scenarios:

* If the jackknife does **not** validate the KS test result:

- If the area test result exceeds the threshold *t*, DPI-based differentiation is detected.

- If the area test result is below the threshold *t*, no DPI-based differentiation is detected.

* If the jackknife **does** validate the KS test result:

- If the area test result does **not** exceed the threshold *t* and the KS test accepts the null hypothesis, no DPI-based differentiation is detected.

- If the area test result exceeds the threshold *t* and the KS test rejects the null hypothesis, DPI-based differentiation is detected.

- In all other cases, the result is considered inconclusive, no DPI-based differentiation is declared.

Summarizing, for the tool to decide that some differentiation occurred, the area test needs to come out positive (excessive area between CDFs) and the KS test needs to be either positive or non-validated by the jackknife method.

## 2.2 Other tools

Note that there have been other differentiation detection tools in the literature, even if most of them are not maintained. We list them here for information, axing the description on the differentiation assertion process.

* In 2008, *Switzerland* [8] was the first noticeable tool, but was not based on a statistical approach. The principle was to hash a number of bits of
<figure><img src="image_4.png" /><figcaption>Figure 3: Wehe decision diagram</figcaption></figure>

packets and investigate if packets were modified, dropped or injected based on the comparison at a server between sent and received content. The method was claimed to be expensive and lacking anonymity.

* Another non-statistical method is adopted by the tool *OONIProbe* [19] which is targeting the blocking of web, instant messaging application, or overlay network traffics. Blocking is expected to happen through Domain Name System (DNS) resolution. A connectivity test is performed by comparing the DNS resolution with a neutral-assumed DNS resolution by Google DNS servers.

* CONNEcT [17] investigates how ISPs can circumvent differentiation detection tools (such as an uncommon Time-to-Live (TTL) in packets). To counter ISP traffic monitoring, CONNEcT makes use of a meta-communication called covered channel; a hidden synchronization allows two machines to insert messages into the legit traffic data with an offset only known from both of them. No statistical analysis is performed.

* In 2009, *NetPolice* [28] was developed to investigate potential differentiation by sending end-to-end packets from several pairs and identifying if more losses occur at a "hop". To measure the loss rate at a hop, probe packets are sent with pre-computed TTL (Time-to-Live) and drops due to null TTL are notified to the sender while those due to differentiation are "silent". Statistically, differences in distributions of losses of end-to-end packets are measured in terms of the type of traffic or of their origin, using a Kolmogorov-Smirnov (KS) statistical test. An uncommon jackknife resampling technique is considered to improve the validity of the test sensitive to small sample sizes, justified by a reference to an abstract [26] (their reference 33), but not much can be found in the literature on that, where bootstrap would rather be expected.
* Still in 2009, NANO [2] was designed to detect differentiation by observing and comparing the performance experienced by different clients, through a central server used for analysis: the goal is to distinguish voluntary discrimination from factors, called confounding factors, including location, time of day, operating system, etc. Confounding factors are grouped into so-called strata and comparisons are made within strata. The method therefore requires identifying the confounding factors and collecting a mass of relevant data for a base of comparison. The stratification is non automated and may be subject to critics, and the resulting statistical test requires justification.

* Glasnost, from the paper published in 2010, [5] investigates pre-identified potential reasons of discrimination. It measures the specific traffic without modification, before replaying it by modifying elements that may cause differentiation (somewhat similarly to Wehe which randomizes the replay). A web interface arbitrarily evaluates the external noise as

$$ \frac{(bitrate_{max} - bitrate_{mean})}{bitrate_{max}} $$

It arbitrarily considers the analysis not possible if that value exceeds 20%. Otherwise, it again arbitrarily considers that differentiation takes place if the difference between flow rates is larger than 50% of the maximum flow rate. A version devoted to mobiles, BonaFide [1], was also developed.

* DiffProbe [10] investigates a potential priority scheduling between two classes of traffic (hence assuming such a classification). The method simultaneously sends the traffic supposedly discriminated (A for application) and a probing flow (P) and statistically compares the results by computing the Kullback-Leibler (KL) divergence $D_{KL}$ between the two distributions of packet delays [11]. It proceeds by discretizing delays into bins, then computing the KL divergence of the two discretized distributions. It then randomly partitions the samples of A delays and computes the corresponding KL divergence between the two resulting empirical distributions. By repeating (bootstrapping) that partitioning, we get a distribution of the KL divergence (between partitions of A samples) and apply the test by looking at the probability that this KL divergence is larger than the one comparing the full distributions for A and P.

* POPI [15] measures packet losses for different packet types. It sends burst of packets to saturate the path and ranks the loss rates between the different types. It then tests unexpected losses using a specific central limit theorem for average normalized ranks.

* Packsen [27] also looks for traffic shaping, by investigating the differences between a reference flow and another one using the suspected differentiation characteristic between a source and a destination. Here, the statistical test considered is Mann-Whitney U test for the inter-arrival of the flows (test
<table><caption>Table 1: Comparison of tools [4].</caption><thead><tr><th>Tool</th><th>Traffic</th><th>Metrics</th><th>Test</th></tr></thead><tbody><tr><td>Switzerland</td><td>any</td><td>packet hash</td><td>comparison</td></tr><tr><td>OONIProbe</td><td>web, DNS, Tor, messaging applications</td><td>DNS resolution, connection success</td><td>comparison</td></tr><tr><td>CONNEcT</td><td>any</td><td>packet loss rate</td><td>no analysis</td></tr><tr><td>NetPolice</td><td>HTTP, BitTorrent, SMTP, PPLive, VoIP</td><td>packet loss rate</td><td>Kolmogorov-Smirnov with jackknife</td></tr><tr><td>NANO</td><td>any</td><td>throughput</td><td>causal inference</td></tr><tr><td>Glasnost</td><td>BitTorrent (can add others)</td><td>throughput</td><td>maximal throughput comparison</td></tr><tr><td>DiffProbe</td><td>Skype, Vonage</td><td>packet loss rate, delay</td><td>Kullback-Leiber</td></tr><tr><td>POPI</td><td>ICMP, FTP, Telnet, POP3, BGP, HTTPS, Fasttrack, Donkey, Gnutella, BitTorrent</td><td>packet loss rate</td><td>ranking, averaging and clustering</td></tr><tr><td>Packsen</td><td>BitTorrent</td><td>inter-arrival packet time</td><td>Mann-Whitney U</td></tr><tr><td>Wehe</td><td>any</td><td>packet loss rate, throughput, delay</td><td>custom Kolmogorov-Smirnov-inspired</td></tr></tbody></table>

of the median of distributions). It is a non-parametric method comparing two distributions based on the ranks of the samples. It is claimed to be robust to outliers and accurate for small sample sizes.

The characteristics of the above tools are summarized in Table 1.

BEREC, the association of European regulators, is also now advertising a network monitoring tool at net-neutrality.tools, monitoring download and upload speed, latency and packet loss. “Nettest” and “Breitbandmessung” are the underlined tools. *Breitbandmessung* is an app (in German) measuring the performance, but not determining if a non-neutral behavior is observed. *Nettest* is a speedtest tool to test the quality of your Internet connection.

BEREC develops applications, with companies such as alladin-IT and zafaco, now available for national implementations. There is an ex-ante determination of minimum, average and maximum values of key performance metrics for end users, and a crowd-sourcing approach is considered but not much is said about statistical decisions. Ofcom, the UK regulator, is also discussing tools (See *Ofcom's network neutrality page*).

## 3 Potential theoretical weaknesses of Wehe requiring further analysis

Wehe acts a bit like a black box: when applying the tool, the user selects an application for which differentiation is tested and the software gives a positive or negative response that has to be trusted without further explanation. Figure 4 provides snapshots of such a process. The decision process and the robustness of the tests used, therefore, require some evaluation, which is limited in the literature.

A theoretical limitation of Wehe's approach is the lack of robust statistical testing when generating responses. Looking further into the implementation of
<figure><img src="image_5.png" /><figcaption>Figure 4: Snapshot of the use of Wehe: choice of apps (left), and test (right)</figcaption></figure>
Wehe's approach raises some questions regarding some statistical decisions that were taken.

First, Wehe runs the KS test on the collected data after eliminating the null values, but keeps them for the other components of the decision process (jackknife and area test). The reasoning for this removal was not discussed. These null values likely originate from the wait time that is implicitly imposed when the packets are exchanged according to the timestamps in the recordings.

Second, the jackknife approach is not commonly used in the statistical literature for handling small samples. Wehe employs that approach, justifying it by it being used in NetPolice, with the threshold $\beta = 0.95$: the original KS test is validated if at least 95% of the resulting tests yield the same conclusion. As for the resampling parameter, Wehe chooses to run $r = 100$ iterations, whereas NetPolice, who previously employed this resampling method, runs $r = 400$ iterations. Jackknife was first introduced in [26] and later used in both of these tools.

Third, while the KS test is theoretically built, the area test is based on heuristics. The area test checks if the normalized area between the CDFs exceeds the threshold $t = 0.1$. Wehe justifies the implementation of the area test through a couple of arguments. In fact, the area test conservatively avoids detecting false positives due to the fact that it is less sensitive to small changes in the distributions, not necessarily attributed to differentiation, than KS. According to Wehe designers' performance metrics, the area test has good accuracy and consistent behavior; the fact remains, however, that the area test does not have a robust theoretical basis.

Wehe's complex decision process of Figure 3 has likely been put in place in order to counter the dependence between observations, which affects the validity of the KS test. We aim to test this hypothesis by comparing with a rigorous statistical approach that relies solely on the KS test in order to provide more reliable results in detecting differentiation.

Another limitation of Wehe's approach lies in its use of recorded packet traces. Those traces are replayed with packet exchanges scheduled according to the original timestamps, in an effort to imitate real TCP behavior between the client and the server. However, this remains a simulation and does not fully replicate the actual dynamics of a live TCP connection. This limitation can impact the quality of the collected data, for example by introducing null values in the measured throughputs. To address this issue, we conducted an experiment using a video being streamed directly on the client side. We collected the resulting traffic and applied the same statistical tests described in the following section. This allowed us to assess how the results may differ when using real traffic instead of pre-recorded traces.
# 4 Numerical tests and comparisons with standard methods

This section describes tests to analyze the issues raised above. While we theorize that Wehe's complex decision rule could be related to the lack of independence in the data that is collected (even if it was not presented as such in the paper); it is also probably related to the willingness to be conservative, that is to minimize false positives, meaning the instances in which differentiation is detected when none is applied. Our work consists of addressing these questions through a pool of tests. We aim to explore an approach that improves KS' performance without relying on heuristics, since KS lies on theoretical grounds while heuristics could be controversial, leading to denials by the accused entities.

To achieve this, we integrate a middlebox into Wehe's architecture to emulate the behavior of network providers. Middleboxes are responsible for mapping network traffic to specific applications and typically include both a classification rule and an associated action. The middlebox we implemented uses the netem queuing discipline [24], which enables us to degrade network performance characteristics - such as throughput, packet loss rate, and delay - either individually or in combination. These degradations are selectively applied to packets based on their classification. We are able, therefore, to explore the parameter space by varying the application trace, the throughput shaping rate, the packet loss rate, and the delay. When relevant, the tests are performed on eleven traces available in Wehe: Amazon, Apple Music, FaceBook Video, Hulu, NBC Sport 2018, NBC Sport 2019, Netflix, Spotify, Twitch, Vimeo and YouTube.

Consequently, we distinguish two separate cases: the first where we do not apply any differentiation parameters (throughput, loss, delay), and the second where we apply a variety of differentiation parameters. For each case, we run a series of tests, once using Wehe's original approach and another using our own approach which consists of relying exclusively on the KS test with a 95% confidence level to reach a final classification.

## 4.1 Numerical tests without differentiation

In this initial series of tests, no differentiation parameters (throughput, loss, delay) are applied. This series of tests focuses on evaluating the false positive rate, that is, the frequency with which differentiation is detected even though none is actually present. The goal is to compare the detection methodologies. We evaluate both Wehe's original approach and a KS-based only approach, which we will refer to as KS approach in what follows. For each approach, we conduct tests in two configurations: one where null throughput values are excluded, and one where they are retained as valid data points.

For each application trace, we perform 30 (independent) iterations for every combination of the test configurations described above, from which we will obtain an estimation of the false positive rate as the proportion of positive tests.
Figures 5a and 5b show the results of these tests. The error bars represent the Wilson Score intervals, which quantify the uncertainty around the estimated detection rate. This type of confidence interval is preferred over the traditional Wald interval, especially in cases involving a small number of trials (in our case, n = 30), as it avoids producing bounds outside the logical range [0, 1] for proportions [3]. The Wilson score interval is centered at:

$$ \hat{p}_{\text{Wilson}} = \frac{\hat{p} + \frac{z^2}{2n}}{1 + \frac{z^2}{n}} $$

with margin of error:

$$ \text{Margin} = \frac{z}{1 + \frac{z^2}{n}} \sqrt{\frac{\hat{p}(1 - \hat{p})}{n} + \frac{z^2}{4n^2}} $$

where $\hat{p} = \frac{k}{n}$ is the observed proportion, k is the number of successes, n is the number of trials and z is the critical value for a 95% confidence level.

When null values are retained, the KS-based approach tends to detect differentiation more frequently than Wehe for some traces (e.g. *NBC Sports 2018* with a false positive rate of 0.17 for the KS approach compared to 0.07 for Wehe's approach), leading to a higher false positive rate in those cases.

However, when the null values are removed, Wehe's performance remains largely unchanged, while the KS approach shows a significant increase in detection-i.e., false positive-rate. This outcome highlights the conservative nature of Wehe's methodology compared to the more rigorous behavior of the KS approach when it comes to identifying differentiation.

## 4.2 Numerical tests with differentiation

In this second series of tests, we consider instances in which differentiation is applied, and therefore introduce an additional layer of parameters to the ones that were considered for the previous series of tests.

More specifically, we vary the throughput, the packet loss and the delay in the following manner:

* *Throughput rate list:* 19500 kbps (100% of the base rate), 17550 kbps, 15600 kbps, 13650 kbps, 11700 kbps, 9750 kbps (50% of the base rate).

* *Delay list:* 0ms, 250ms, 500ms, 750ms, 1000ms

* *Loss list:* 0%, 1%, 2%, 3%, 4%, 5%, 6%

The differentiation is applied as a combination of these values in an attempt to cover the parameter space as fully as possible. For each combination, we run only one iteration of the test (for computational reasons).

The performance criterion for this set of tests is the detection rate across the entire differentiation parameter space for each application trace. The results are shown in Figures 6a and 6b.
<figure><img src="image_6.png" /><figcaption>Figure 5: Test results - No differentiation</figcaption></figure>
When null throughput values are retained, the KS-based approach consistently outperforms Wehe's original method, detecting differentiation more frequently across all tested traces and parameter combinations. The detection rate for KS varies from 74.26% (*Amazon*) to 97.47% (*NBC Sports 2018*) with Wehe consistently having slightly inferior detection rates for each trace, in average a detection rate more than 10% smaller.

After removing the null values, this trend largely persists. However, detection rates increase noticeably for both approaches across all traces. Notably, Wehe's detection rate improves significantly, becoming nearly comparable to that of the KS approach. In the case of Apple Music, Wehe's method even significantly outperforms the KS-based approach (80.86% of detection rate as opposed to 55.71%).

## 4.3 Time interval for computing throughput values

The underlying issue that Wehe is trying to address through its complex decision rule is the (assumed) lack of independence in the observations that are being collected. KS is a statistical test with theoretical grounds that requires independence of the observations. In the absence of this prerequisite condition, KS does not perform as well as expected. As an alternative to adding steps (like cross-validation or the area test) that make the decision process more conservative and validate the KS test, we decided to examine the data and the process of collection more closely.

In Wehe, the average throughput is collected every 0.22 seconds, that is, we compute the volume of data received over periods of 0.22 seconds divided by that duration. That short time interval could possibly cause a dependence between the samples. In what follows, we examine the effect that varying that interval could have on the independence of the samples, and by extension the validity of the KS test.

Our reasoning follows loosely from the batch means method used for [23]. Batch means, in theory, consists of essentially organizing our observations into batches as seen in Figure 7, computing the average mean of each batch, and considering those means as our observations instead. This, consequently, increases the independence between the observations since the impact of (dependent) successive values is decreased by the averaging over the batch.

Adapting this to the current context means that we are looking to vary – more specifically, increase – the durations between the times of collection in order to see if it has any effect on the level of dependence between the observations. This approach allows us to have a theoretically-based confidence interval under the appropriate conditions which can ensure independence of observations (in our case, average throughputs).

### 4.3.1 Independence analysis on a Wehe trace

We first test the required independence assumption for the KS test when a time interval of 0.22 seconds is used to get observations; on the Netflix trace
<figure><img src="image_7.png" /><figcaption>(a) With null values</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) Without null values</figcaption></figure>

<caption>Figure 6: Test results - Differentiation</caption>
<figure><img src="image_9.png" /><figcaption>Figure 7: Batch means concept [23]</figcaption></figure>

as it is among the longest traces in the application, with a total run time of approximately 30 seconds, and therefore easier to manipulate to get the largest possible number of observations.

We use the Ljung-Box test to evaluate the independence of the observations. This statistical test allows us to verify if a time series presents a significant autocorrelation on multiple lags [14]. Under the null hypothesis that the data is not correlated, the Ljung-Box test considers the following statistic:

$$Q(m) = n(n + 2) \sum_{k=1}^{m} \frac{\rho_k^2}{n - k}$$

where:

* n is the sample size,

* $\rho_k$ is the autocorrelation on lag k,

* m is the number of lags considered.

The test rejects the null hypothesis, in other words, it considers the data as correlated if for a significance level $\alpha$, the statistic Q strictly exceeds $\chi_{1-\alpha,h}^2$, the $(1-\alpha)$-quantile of the chi-squared distribution with h degrees of freedom.

The results with a batch size of 0.22 are provided in Figures 8 and 9.

In Figure 8, we observe the Ljung-Box statistic for up to 10 lags as well as the associated p-values. For a p-value smaller than 0.05, we consider the possibility of a non-zero correlation within the first k lags. We can also observe that the autocorrelation values for the first few lags are significantly high as can be seen in Figure 9.

We represent in Figure 9 the throughput over time for one replay run, as well as the autocorrelation plots. The autocorrelation plot represents the autocorrelation
value for each of the corresponding lags as well as a 95% confidence interval. We
can effectively see that there is significant autocorrelation for the first lags.

<figure><img src="image_10.png" /><figcaption>Figure 8: Independence test results and autocorrelation values</figcaption></figure>

This highlights the interest of investigating the impact of larger batch sizes.
Though, it brings us to a related point of concern: the length of the recorded
traces. The traces in the Wehe app do not exceed 30 seconds, hence using
significantly larger batch sizes would lead to extremely small samples.

4.3.2 Analysis on an own-built video trace

As a way to counteract the above limitation on the number of observations, we
ran additional tests by streaming a video on the client side with a large run
time, then implementing the previously established tests. The time intervals we
consider range from 0.22 seconds to 16 seconds as follows: 0.22, 1, 4, 10, and 16.

Figures 10 and 11 compare the independence of observations between the
two extreme values of time interval: 0.22s and 16s. An improvement is observed,
as expected.

At an interval of 16 seconds, the throughputs show clear independence,
therefore, this is the interval that we test in what follows.

In order to see the effect of changing the interval on the detection of
differentiation, we run tests similar to what was done in Sections 4.1 and 4.2:
We run the Wehe approach with a time interval of 0.22 seconds for 5 minutes,
and compare it to the KS approach which was done with a time interval of 0.22,
1, 4, 10, and 16 seconds for the same trace of 5 minutes.

To investigate false positive rates, we first run 10 iterations of the test with no
differentiation. As can be seen in Figure 12a, Wehe does not encounter any false
positive, being very conservative. KS on the other hand, has a non-negligible
<figure><img src="image_11.png" /><figcaption>Figure 9: Throughputs and autocorrelation</figcaption></figure>
<figure><img src="image_12.png" /><figcaption>Figure 10: Independence test results and autocorrelation values</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 11: Throughputs and autocorrelation</figcaption></figure>
false positive rate, except if we reach a long time interval to compute values, that is 16s.

<figure><img src="image_14.png" /><figcaption>Figure 12: Test results</figcaption></figure>

As for instances in which differentiation is applied, we consider the following parameters:

* *Throughput rate list:* 19500 kbps (100% of the base rate), 14625 kbps, 9750 kbps (50% of the base rate).

* *Delay list:* 0ms, 500ms, 1000ms

* *Loss list:* 2%, 6%

We display in Figure 12b the detection rate over all above combinations of differentiation parameters. Interestingly, All detection rates are close, but Wehe perform similarly to the case of best KS case, with a 88.89% detection rate.

# 5 Conclusions

Wehe is a network neutrality detection tool giving positive or negative responses to the question “does my ISP differentiate service for the tested application?”, using recorded traces for some of the most used applications. Though, Wehe acts a bit like a black box to be trusted; our goal in this paper was to investigate its validity and potentially to propose variations with more theoretical grounds, since heuristics could be contested in court.

We have illustrated that Wehe is very conservative, hence limiting the false positive rate, but that the test is non standard. We compare it with the standard Kolmogorov-Smirnov test, for which we choose to collect throughput data over sufficiently large intervals to ensure independence and therefore test validity. Despite that, our preliminary results suggest that Wehe performs as least as well as our more standard method, and even better as it comes to false positives, implying the tool is quite reliable.

We plan to perform more extensive testing of the tool in a near future.
# Acknowledgments

The authors acknowledge the support of Inria's Regalia initiative.

# References

[1] V. Bashko, N. Melnikov, A. Sehgal, and J. Schönwälder. Bonafide: A Traffic Shaping Detection Tool for Mobile Networks. 2013 IFIP/IEEE International Symposium on Integrated Network Management, May 2013.

[2] M. Bin Tariq, M. Motiwala, N. Feamster, and M. Ammar. Detecting Network Neutrality Violations with Causal Inference. CoNEXT '09: Proceedings of the 5th International Conference on Emerging Networking Experiments and Technologies, pages 289-300, 2009.

[3] Lawrence D. Brown, T. Tony Cai, and Anirban DasGupta. Interval estimation for a binomial proportion. Statistical Science, 16(2):101-117, 2001.

[4] X. Castoreo, P. Maillé, and B. Tuffin. Weaknesses and Challenges of Network Neutrality Measurement Tools. In Proc. of 16th IEEE International Conference on Network and Service Management (CNSM), Virtual Conference, April 2020.

[5] M. Dischinger, M. Marcon, S. Guha, K.P. Gummadi, R. Mahajan, and S. Saroiu. Glasnost: Enabling End Users to Detect Traffic Differentiation. Proc. of USENIX Symposium on Networked Systems Design and Implementation, 2010.

[6] J. Durbin. Distribution Theory for Tests Based on the Sample Distribution Function. Society for Industrial and Applied Mathematics, 1973.

[7] R. Easley, H. Guo, and Jan Krämer. From network neutrality to data neutrality: A techno-economic framework and research agenda. SSRN Electronic Journal, 01 2015.

[8] P. Eckersley. Switzerland Design. https://www.eff.org/files/2018/06/21/design.pdf, May 2008.

[9] T. Garrett, L. E. Setenareski, L. M. Peres, L. C. E. Bona, and E. P. Duarte. Monitoring network neutrality: A survey on traffic differentiation detection. IEEE Communications Surveys Tutorials, 20(3):2486-2517, 2018.

[10] P. Kanuparthy and C. Dovrolis. Diffprobe: Detecting ISP service discrimination. In Proc. of IEEE INFOCOM, San Diego, CA, USA, 2010.

[11] S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79-86, March 1951.
[12] T.M. Lenard and R.J. (Eds.) May. *Net Neutrality or Net Neutering: Should Broadband Internet Services be Regulated*. Springer, 2006.

[13] Fangfan Li, Arian Akhavan Niaki, David Choffnes, Phillipa Gill, and Alan Mislove. A large-scale analysis of deployed traffic differentiation practices. *Proceedings of the ACM Special Interest Group on Data Communication*, page 130–144, 2019.

[14] G. M. Ljung and G. E. P. Box. On a measure of lack of fit in time series models. *Biometrika*, 65(2):297–303, 08 1978.

[15] G. Lu, Y. Chen, S. Birrer, and F.E. Bustamante. POPI: A User-Level Tool for Inferring Router Packet Forwarding Priority. *IEEE/ACM Transactions on Networking*, 18(1):1–14, February 2010.

[16] P. Maillé and B. Tuffin. *From Net Neutrality to ICT neutrality*. Springer, 2022.

[17] A. Maltinsky, Giladi R., and Y. Shavitt. On Network Neutrality Measurements. *ACM Transactions on Intelligent Systems and Technology*, 8(4), May 2017.

[18] A. Molavi Kakhki, A. Razaghpanah, A. Li, H. Koo, R. Golani, D. Choffnes, P. Gill, and A. Mislove. Identifying Traffic Differentiation in Mobile Networks. *Proceedings of the 2015 Internet Measurement Conference*, pages 239–251, October 2015.

[19] Network Neutrality Squad. Open Observatory of Network Interference. https://ooni.org. Accessed March 5, 2020.

[20] NEU-SNS. Wehe: Detect traffic differentiation in mobile networks. https://github.com/NEU-SNS/wehe-py3, 2020. Accessed: 2025-04-29.

[21] Northeastern University Wehe Project. Wehe. https://play.google.com/store/apps/details?id=mobi.meddle.wehe on Google Play Store. Accessed March 5, 2020.

[22] A. Odlyzko. Network neutrality, search neutrality, and the never-ending conflict between efficiency and fairness in markets. *Review of Network Economics*, 8(1):40–60, 2009.

[23] Manuel D. Rossetti. *Simulation Modeling and Arena*. Wiley Publishing, 2nd edition, 2015.

[24] The Linux Foundation. netem. https://wiki.linuxfoundation.org/networking/netem. last accessed 23 Oct 2020.

[25] Tim Wu. Network Neutrality FAQ. http://www.timwu.org/network_neutrality.html. Accessed February 10, 2020.
[26] John W. Tukey. Bias and confidence in not-quite large samples. *Annals of Mathematical Statistics*, 29(2):614–623, 1958.

[27] U. Weinsberg, A. Soule, and L. Massoulie. Inferring Traffic Shaping and Policy Parameters using End Host Measurements. *Proceedings - IEEE INFOCOM*, May 2011.

[28] Y. Zhang, Z. Morley Mao, and M. Zhang. Detecting traffic differentiation in backbone isps with netpolice. In *Proc. of 9th ACM SIGCOMM conference on Internet measurement (IMC)*, pages 103–115, Chicago, IL, USA, Nov 2009.