Social Benchmarking of QoS & QoE
in Cellular Data Networks
Hirochika Asai

Kensuke Fukuda

Hiroshi Esaki

The University of Tokyo
panda@hongo.wide.ad.jp

NII / PRESTO JST
kensuke@nii.ac.jp

The University of Tokyo
hiroshi@wide.ad.jp

Abstract—We developed a benchmarking tool called NetBenchto measure the quality of service (QoS) and quality of experience(QoE) in cellular data networks and to share them among usersof the tool. This tool has a user interface that displays thesummarized QoS and QoE on a map to motivate users to share
their QoS and QoE. We have been continuously collecting QoS
related metrics and a QoE related metric from this tool. We
characterize the QoS and QoE of three cellular data network carriers (one is a CDMA2000 operator and the others are UMTS/WCDMA operators supporting HSDPA/HSUPA) in Japan from thecollected data for eight months. We demonstrate that round trip
time (RTT) among other QoS related metrics such as signal
strength and packet loss rate is one of the metrics that explains
the performance of TCP throughput. We also demonstrate thatthe packet loss rate and RTT of the QoS related metrics affects
the QoE. These overall results of QoS and QoE indicate that thecharacteristics of QoE is explained by QoS, especially by RTT.
Index Terms—Cellular data network, 3G, quality of service,quality of experience

I. I NTRODUCTION
Cellular data networks have been globally deployed, andthe traffic volume in the cellular data networks is dramaticallyincreasing [1]. Therefore, the cellular data networks are an
essential component of the Internet. Understanding the qualityof service (QoS) and quality of experience (QoE) in thecellular data networks is crucial for network design and trafficengineering as well as performance evaluation. From theviewpoint of subscribers of the cellular data networks, it isalso important to know the actual service level from QoS andQoE because the cellular data network service is a best effort
service, and consequently the actual service level is often very
different from the nominal service level announced by cellularcarriers. However, it is not easy to compare such metrics overmultiple operators though each likely has them as internal data.
We developed a social benchmarking tool called NetBenchfor smartphones to measure both the QoS and QoE in cellular
data networks and to share them among the users of thetool [2]. From this tool, we have been collecting QoS related
metrics such as TCP throughput, signal strength, ICMP packet
loss rate, and round trip time (RTT) based on the active
measurement approach, and a QoE related metric that is user’s
vote (good or bad).
In this paper, we introduce the developed tool and thenpresent the characteristics of the QoS and QoE of threecellular data network carriers (one is a CDMA2000 operator
and the others are UMTS/W-CDMA operators supporting

HSDPA/HSUPA) in Japan from the collected data for eightmonths. As for the QoS related metrics, we demonstrate that
the RTT is one of the metrics that explains the TCP throughput. We observe less correlation between signal strength,
ICMP packet loss rate, and TCP throughput for each carrier.
On the other hand, the correlation between RTT and TCP
throughput is stronger than any correlations between TCP
throughput and other metrics for every carrier. In particular, weobserve a strong correlation between RTT and TCP throughput
for one carrier using UMTS/W-CDMA. As for the QoE related
metric, we demonstrate that the RTT and the packet loss rate
mainly affect the QoE. We show that the characteristics of QoE
is explained by QoS, especially by RTT, from these overall
results.
The contributions of this paper are twofold: 1) We developed
a social benchmarking tool to measure QoS and QoE. 2)
We show the characteristics of the QoS and QoE and theircorrelation in cellular data networks from the data collectedfrom the developed tool.
II. R ELATED W ORKSThe QoS and QoE analysis in the cellular data networks has
not been well researched in prior works. Halepovic et al. [3]and Williamson et al. [4] focused on the user mobility suchas cell changes in a cellular data network and they modeled
the temporal characteristics of the user mobility from theirmeasurement in CDMA2000 network. However, they do not
analyze the performance at the actual service level of thecellular data network.
Romirer-Maierhofer et al. [5] reported their TCP RTT
measurement results in GPRS/EDGE and UMTS/HSxPA networks. Balasubramanian et al. [6] measured the QoS such as
TCP/UDP throughput and packet loss rate in 3G and WiFi
networks. However, they have not focused on the QoE of users
and the correlation between QoS and QoE.
Huang et al. [7] measured the high-level performance ofsmartphones such as pervasive Web browsing while taking
into account the differences of the hardware and software ofdevices. The high-level performance on which they focus issimilar to the QoE, but they do not analyze the correlation
between low-level QoS such as throughputs and QoE.
Joel et al. [8] analyzed the performance of cellular and802.11 WiFi networks at metropolitan areas using a large
scale dataset obtained at a throughput measurement site. Their2

approach is similar to ours in terms of using crowd-sourced
data. However, the have not collected the QoE of users andanalyzed the correlation between QoS and QoE.
Shafiq et al. [9] modeled the characteristics of Internet trafficof cellular devices by analyzing the flow level traffic datacollected at cellular data network carrier’s core network. Xu etal. [10] have also focused on the cellular data network carriers’
wired network infrastructure, and analyzed the characteristicsof routing in the carrier’s network. However, their focus iscarrier’s core network only, and consequently, they could notachieve to analyze the characteristics and correlation between
the QoS and QoE through wireless and wired networks ofcellular data networks.
Jin et al. [11] has developed a smartphone application
to collect troubles of cellular data networks from mobile
customers. Their approach is similar to our QoE collection, butthey do not analyze the characteristics and correlation between
the QoS and QoE.
The QoE has gained great interest and characterizing thecorrelation between QoS and QoE become increasingly important. Schatz et al. [12] focus on both the QoS and QoE.
They conducted lab and field trials to measure the QoS and
end-user QoE. Their focus is similar to ours, but their trials
do not highlight the correlation between the low-level QoS
related metrics such as RTT and QoE.
III. P RELIMINARY(a) QoS
Fig. 1.

(b) QoE

Screenshots of the user interface to browse shared QoS/QoE posts

TABLE IT HE BREAKDOWN OF THE Q O S AND Q O E POSTS BY CARRIER

QoSCarrier

QoE+1 (good) -1 (bad)KDDI
Softbank
docomoOthers

791
316
173
5

419
22
3
3

63
18
3
2

WiFi

158

45

27

A. Social benchmarking tool
We developed a social benchmarking tool NetBench [2] for
iPhones and iPads [13] supporting iOS version 5 or later tomeasure the QoS and QoE in cellular data networks and toshare them among the users of the tool at the user interface
on the device and the Web site. This tool supports two
measurements: 1) QoS and 2) QoE.
The former, QoS measurement, is implemented as the active
measurement. It first sends 21 ICMP echo request packets
to a measurement server in Tokyo and receives ICMP echo
reply packets with 3 second timeout to measure the ICMP
packet loss rate and the minimum, median, and maximum
RTT. Next, it measures the average download throughput
from the measurement server for 10 seconds using TCP withan unprivileged port number. Likewise, it then measures theaverage upload throughput to the measurement server for 10
seconds using TCP with an unprivileged port number. At theend of this procedure, it records the signal strength1 , andgeolocation information of the device.
The latter, QoE measurement, is the combination of user’s
vote and active measurement. Users can choose either +1(good) or -1 (bad) at the graphical user interface of thedeveloped tool according to their feelings (i.e., QoE). Aftera user chooses the alternative of good or bad, it records
the signal strength and geolocation information. It then sends
10 ICMP echo request packets to a measurement server and
1 The unit of signal strength is unknown but it is the value as is obtained
through iOS’s API.

receives ICMP echo reply packets with 3 second timeout tomeasure the ICMP packet loss rate and the minimum, median,and maximum RTT.
The posts from the users of the same carrier are sharedthrough the user at the user interface of the tool on a device.
We present the screenshots in Fig. 1. Figure 1(a) is thescreenshot showing the QoS that places pins with the averagethroughputs on the map. Figure 1(b) is the screenshot showing
the QoE that places tiles with the average QoE score in the
Hue color space from red (bad) to blue (good) on the map.
Similarly, shared posts from users of each carrier can bebrowsed at the Web interface. These interfaces motivate users
to share their QoS and QoE with other users.

B. Collected Data
We obtained 1443 (shared) posts of QoS and 605 postsof QoE in eight months (from mid-March to mid-November,2012). The breakdown of the QoS and QoE posts by carrier isshown in TABLE I. The majority of the posts are from threecarriers in Japan, and consequently, we focus on these threecarriers excluding measurements and posts using WiFi in thispaper. Note that KDDI is CDMA2000 operator, and Softbank
and docomo are UMTS/W-CDMA operators supporting HSDPA/HSUPA.

3

100

Up [Mbps]

10

1

0.1

0.01
KDDI
Softbank
docomo0.001
0.01

Fig. 2.

0.1

1Down [Mbps]

10

100

TCP download versus upload throughputs

IV. A NALYSIS R ESULTS
A. Quality of Service
We focus on the QoS related metrics and analyze thecorrelations between these metrics in this subsection.
We first present a scatter plot of the average TCP download
throughput versus the average TCP upload throughput in
Fig. 2. We expected a strong positive correlation between
the download and upload throughputs because the wirelessradio path environment is almost equal. However, the actual
correlations between the download and upload throughputs
are not much strong, as the correlation coefficients between
the download and upload throughputs with the logarithmic
scale of KDDI, Softbank, and docomo are 0.510, 0.675, and
0.341, respectively. This is because the download links arelikely more congested than the upload links, and consequently,
the download throughput is dispersed more than the upload
throughput. Another insight from Fig. 2 is that the correlation
coefficients of KDDI and docomo are relatively smaller than
Softbank. A plausible reason of this is the difference of theconfiguration and traffic engineering policy of carriers; KDDIand docomo seem to limit the maximum speed to keep fairness
while the configuration of Softbank is based on best effort.
We then show scatter plots of the signal strength versus
average TCP download/upload throughput in Fig. 3. Here, we
exclude the results of docomo from this figure because the
API fails to obtain the signal strength through the operating
system when it uses a SIM card of docomo. We note that wecannot directly compare the signal strength values between
KDDI and Softbank because they use different 3G standards.
The signal strength is commonly used to indicate the QoS anddisplayed on the user interface of devices. However, Fig. 3
shows that the signal strength does not affect the throughput.
We confirm very small positive correlation between the signal
strength and the average TCP download/upload throughput at
KDDI; the correlation coefficients between the signal strengthand the average TCP download/upload throughput with thelogarithmic scale are 0.031 and 0.137, respectively.
Figure 4 shows the correlation between the ICMP packet

loss rate and the average TCP download/upload throughput
with the logarithmic scale. It is commonly known that packet
loss results in remarkable TCP performance degradation due
to retransmission and a shrink of the sliding window. However,this result shows that packet loss does not cause remarkable
TCP performance degradation. We discuss the reason of thisresult here; in the developed tool, we setup the timeout in the
ICMP packet loss measurement to 3 seconds, meaning thatpackets exceeding 3 seconds of RTT are counted as packet
loss. Therefore, most of the packets that are detected as packet
loss are not actually dropped but properly delivered between
smartphones and the Internet by being buffered at smartphones
or intermediate devices in cellular data networks for several
seconds. We observe negative correlation between the packet
loss rate and the average TCP download/upload throughput
with the logarithmic scale. Two reasons can be given for thiscorrelation: 1) actual packet drops cause the lower throughput,and 2) larger RTT of packets results in lower throughput.
Figure 5 presents the absolute correlation between the
median of ICMP RTT with the logarithmic scale and theaverage TCP download/upload throughput with the logarithmic scale. We find that the correlation between RTT and
TCP throughput is stronger than any correlations between the
average TCP download/upload throughput and other metrics
described above for every carrier. In particular, we observea strong negative correlation between the median of RTT
and the average TCP download/upload throughput with thelogarithmic scale for Softbank; the correlation coefficients fordownload and upload are −0.507 and −0.601, respectively.
We summarize the correlation coefficients between the QoS
related metrics in Fig. 6. These matrices demonstrate that the
RTT is one of the metrics that explains the performance of
TCP throughput most especially in one operator. The signal
strength that is displayed to smartphone users as a quality
indicator has small correlation with the other metrics. The
ICMP high packet loss rate does not result in the noticeable
throughput degradation, but this is because most of packets
counted as packet loss are detected as being dropped due totimeout and they are properly delivered explained above.
B. Quality of Experience
In this subsection, we analyze the QoE posts and thendiscuss the characteristics between the QoS and QoE.
The probability density function (PDF) and complementary
cumulative distribution function (CCDF) of the median RTT
by QoE are presented in Fig. 7. This figure shows that users
experience bad quality in the cellular data networks at larger
RTT. Specifically, 200 milliseconds for the median RTT is theborderline where users experience bad quality. The RTT has acorrelation with the TCP throughput as shown in Section IV-A,
but we do not observe a shrink of the TCP download/upload
throughput around at 200 milliseconds of the median RTT(Fig. 5). Therefore, the RTT is the dominant QoS parameterthat affects the QoE. Since the major carrier for the QoE
posts is KDDI as shown in TABLE I, we next focus on theKDDI. In Fig. 5, the points of KDDI are concentrated less

4

100

100
KDDI
Softbank

KDDI
Softbank
10
Throughput [Mbs]

Throughput [Mbs]

10

1

0.1

1

0.1

0.01

0.01

0.001

0.001
0

20

40

60
80
Signal strength

100

120

0

20

40

(a) Download
Fig. 3.

60
80
Signal strength120(b) Upload
Signal strength versus TCP download/upload throughput

100

100
KDDI
Softbank
docomoKDDI
Softbank
docomo10
Throughput [Mbs]10
Throughput [Mbs]

100

1

0.1

1

0.1

0.01

0.01

0.001

0.001
0

10

20

30

40

50
60
Loss [%]

70

80

90

100

0

10

20

(a) Download

40

50
60
Loss [%]

70

80

90

100

(b) Upload
Packet loss rate versus TCP download/upload throughput

than 200 milliseconds of the median RTT and the shrink of
the TCP download/upload throughput cannot be observed aswell. This implies that the median RTT becomes less than 200
milliseconds at the stable network environment that brings a
good QoE to users, but the throughput does not significantly
affect the QoE.
Figure 8 shows the CCDF of packet loss rate by QoE. As
shown in Section IV-A, the packet loss rate is highly correlated with the RTT due to the timeout of the measurement.
Therefore, the packet loss rate as well as the RTT affects
the QoE. Note that the timeout of the ICMP packet loss
measurement in QoE is set to the same value, 3 seconds, asthat in QoS. Therefore, as described in the QoS result, packets
that are detected as packet loss shall not be actually dropped
but properly delivered between smartphones and the Internet
by being buffered at smartphones or intermediate devices incellular data networks for several seconds. To support thisresult, we will also evaluate the TCP packet loss rate as wellas the ICMP packet loss rate in the future experiments as
discussed in Section V.

1
+1
-1

0.9
0.8
0.7
0.6
CCDF

Fig. 4.

30

0.5
0.4
0.3
0.2
0.1
0
0

20

Fig. 8.

40
60
Packet loss rate

80

100

CCDF of packet loss rate by QoE

We do not measure the correlation between the QoE andthroughput in this paper, but the results shown above demonstrate that the RTT is the dominant QoS parameter that affects

5

100

100
KDDI
Softbank
docomoKDDI
Softbank
docomo10
Throughput [Mbs]

Throughput [Mbs]

10

1

0.1

0.01

1

0.1

0.01

0.001

0.001
10

100
1000
Median RTT [ms]

10000

10

100
1000
Median RTT [ms](a) Download
Fig. 5.

(b) Upload
The median of RTT versus TCP download/upload throughput

1
Max RTT

1
Max RTT

0.8

Median RTT
Min RTT

0.6

Loss
0.4

Up

0.2

Down

0.8

Median RTT
Min RTT

0.6

0
RTT T
al s
RTT ian RT
Sign Los Min Med Max(a) KDDI

0.8

Median RTT
Min RTT

0.6

Loss
0.4

Signal
Up

0.2

Down

n
Dow Up1
Max RTT

Loss

Signal

0.4Signal
Up

0.2

Down

n
Dow Up

0
RTT T
al s
RTT ian RT
Sign Los Min Med Maxn
Dow Up(b) Softbank
Fig. 6.

10000

0
RTT T
al s
RTT ian RT
Sign Los Min Med Max(c) docomo

The absolute values of correlation coefficients between the QoS related metrics

the QoE. Note that the throughput is correlated with the RTT,
and consequently, the correlation between the QoE and thethroughput is also conceivable.
V. D ISCUSSION
Differences of hardwares and operating systems: In thispaper, we do not take into account the differences of hardwares
and operating systems. We confirm that few differences of operating systems exist because the developed tool only support
iOS version 5 or later. As for the hardwares, we admit thatdifferences of hardwares shall affect the QoS as pointed out
in Ref. [7]. However, we could not collect enough numberof samples for various hardwares in this paper and our QoS
measurement is not a heavy task consuming the CPU resource
although the QoE shall be affected by hardware spec, so we
do not take into account the difference of hardwares. We aredeveloping the tool for another platform (Android [14]). We
will evaluate the QoS and QoE for different hardwares andoperating systems in future.
Collected metrics: We have collected TCP throughput,
signal strength, ICMP packet loss rate, and ICMP RTT for
QoS related metrics, and user’s vote (good or bad) for a
QoE related metric. Collecting the QoE related metric is a
step forward to understand the relationships between QoS

and QoE. However, we have not collected other considerable
and possibly significant QoS related metrics such TCP packet
loss rate, TCP RTT, and DNS lookup delay. In this paper,
we have used ICMP RTT and ICMP packet loss rate insteadof TCP RTT and TCP packet loss rate although it is known
that ICMP might not be good to explain the RTT and packet
loss rate for data packets (e.g., TCP packets) due to low
priority configuration for ICMP packets. This is because of
the difficulty of collecting TCP RTT and packet loss rateon smartphones due to their API restrictions. However, themetrics based on ICMP still indicate coarse grained view of
the network status as shown in Fig. 4, Fig. 5, and Fig. 7. We
started the experiment with the small number of metrics thatare easy to be implemented to show that collecting the QoS
and QoE related metrics are significant to understand the QoS
and QoE. We plan to collect other metrics in the future versionof the developed tool to discuss more detailed characteristics
between the QoS and QoE.
Middleboxes: It is well known that some middleboxes
such as carrier grade NAT gateways, accelerators, and transparent Web proxy servers are usually deployed in cellular
data networks [15]. In this paper, we use unprivileged ports(above 1023) for TCP throughput measurements to avoid theinterception of transparent Web proxy servers. Likewise, we

6

0.16

1
+1
-1

0.14

+1
-1

0.9
0.8

0.12
0.7
0.6
CCDF

PDF

0.1
0.08

0.5
0.4

0.06

0.3
0.04
0.2
0.02

0.1

0

0
10

100

1000
Median RTT [ms]10000

10000010100(a) PDF

1000
Median RTT [ms]10000

100000(b) CCDF
Fig. 7.

PDF and CCDF of the median RTT by QoE

do not use TCP RTT and packet loss rate but ICMP RTT
and packet loss rate to ignore TCP accelerators. However,
we can consider that the QoE is affected by the acceleration
mechanisms of these middleboxes. We will evaluate the impactof these middleboxes by measuring application-level (e.g.,
HTTP) goodputs and TCP RTT and packet loss rate as well.
Use of time and geolocation data: We have collected timeand geolocation information with the QoS and QoE posts,
but we have not analyzed the time and geolocation data inthis paper because the number of samples is not enough tostatistically analyze the QoS and QoE by hour and geolocation
at the moment. We think the QoS and QoE vary by hour, andthe geolocation data have the potential to shed light on the
QoS and QoE at the congested environment in a metropolitan
area. We will analyze time and geolocation data when we canobtain enough number of samples.
VI. C ONCLUSION
We developed a social benchmarking tool for smartphones
to measure the QoS and QoE in cellular data networks andshare them among the users of the tool. In this paper, we
presented the characteristics of the QoS and QoE of threecellular data network carriers (one is a CDMA2000 operator and the others are UMTS/W-CDMA operators supporting HSDPA/HSUPA) in Japan from the collected data. We
demonstrated that RTT, rather than other QoS related metrics
such as signal strength and packet loss rate, was one of themetrics that explains the performance of TCP throughput. We
also demonstrated that the packet loss rate and RTT of the
QoS related metrics affected the QoE. We showed that thecharacteristics of QoE was explained by QoS, especially by
RTT, from these overall results.
Unfortunately, the number of available posts for QoE wascurrently limited. So, we will improve the user interface ofthe tool to attract users and collect more QoS and QoE posts.
We also plan to add some metrics related to QoS such as TCP
RTT, TCP packet loss rate, HTTP goodput, UDP throughput,and delay in DNS.

ACKNOWLEDGMENT
We thank anonymous users for sharing their QoS and QoE.
R EFERENCES[1] “Cisco visual networking index: Global mobile data traffic
forecast update, 20112016,” http://www.cisco.com/en/US/solutions/collateral/ns341/ns525łlowbreak/ns537/ns705/ns827/white paper c11520862.html.
[2] Scyphus Solutions Co. Ltd., “Network Social Benchmark,”
http://netbench.scyphus.co.jp/.
[3] E. Halepovic and C. Williamson, “Characterizing and modeling user
mobility in a cellular data network,” in ACM PE-WASUN ’05. ACM,
2005, pp. 71–78.
[4] C. Williamson, E. Halepovic, H. Sun, and Y. Wu, “Characterization ofCDMA2000 cellular data network traffic,” in IEEE LCN ’05, 2005, pp.
Z000 –719.
[5] P. Romirer-Maierhofer, F. Ricciato, A. DAlconzo, R. Franzan, and
W. Karner, “Network-wide measurements of TCP RTT in 3G,” Traffic
Monitoring and Analysis, pp. 17–25, 2009.
[6] A. Balasubramanian, R. Mahajan, and A. Venkataramani, “Augmenting
mobile 3G using WiFi,” in MobiSys ’10. ACM, 2010, pp. 209–222.
[7] J. Huang, Q. Xu, B. Tiwana, Z. M. Mao, M. Zhang, and P. Bahl,“Anatomizing application performance differences on smartphones,” inMobiSys ’10. ACM, 2010, pp. 165–178.
[8] J. Sommers and P. Barford, “Cell vs. WiFi: On the performance of metroarea mobile connections,” in ACM IMC ’12. ACM, 2012, pp. 301–314.
[9] M. Z. Shafiq, L. Ji, A. X. Liu, and J. Wang, “Characterizing and modeling internet traffic dynamics of cellular devices,” in ACM SIGMETRICS’11. ACM, 2011, pp. 305–316.
[10] Q. Xu, J. Huang, Z. Wang, F. Qian, A. Gerber, and Z. M. Mao, “Cellular
data network infrastructure characterization and implication on mobile
content placement,” in ACM SIGMETRICS ’11. ACM, 2011, pp. 317–
328.
[11] Y. Jin, N. Duffield, A. Gerber, P. Haffner, W.-L. Hsu, G. Jacobson,
S. Sen, S. Venkataraman, and Z.-L. Zhang, “Large-scale app-based
reporting of customer problems in cellular networks: potential andlimitations,” in ACM SIGCOMM W-MUST ’11. ACM, 2011, pp. 13–18.
[12] R. Schatz and S. Egger, “Vienna surfing: assessing mobile broadband
quality in the field,” in ACM SIGCOMM W-MUST ’11. ACM, 2011,
pp. 19–24.
[13] “Apple Inc.” http://www.apple.com/.
[14] “Android,” http://www.android.com.
[15] Z. Wang, Z. Qian, Q. Xu, Z. Mao, and M. Zhang, “An untold storyof middleboxes in cellular networks,” in ACM SIGCOMM ’11. ACM,
2011, pp. 374–385.