Malformed EAP packets:
investigation and analysis
Josh Howlett (Federated Solutions / Internet2)
josh@federated-solutions.com
9th June, 2023

Introduction• This presentation summarises an issue that has effected the Internet2
eduroam service since early 2023
• We have identified the root cause and taken steps to reduce its impact• However, the issue is widespread across the eduroam network, so it isimportant that NROs (and RADIUS proxy operators in general) are aware• The Internet2 eduroam service was probably first effected because of thesize of the US federation• The goals of this presentation are to• socialise the issue among eduroam NROs, and other RADIUS proxy operators• describe how we discovered the cause• explain how Internet2 is mitigating the impact, and• pose some broader questions raised by this issue

Initial reports from Service Providers• The first indication of anissue were two tickets
raised by two Service
Providers on February 14th
• Both institutions reported
a very similar issue:
visitors were frequently
unable to authenticate fora period of a few hours• Our FreeRADIUS logs showed an error of “Failed allocating Id for proxied request”

Mysterious proxy error• There are multiple
instances of a second error
message “Failed to insert
request into the proxy list”
in our proxy logs
• The errors are notassociated with any specific
SPs or IDPs

FreeRADIUS proxy• FreeRADIUS uses a structure
named proxy_list to trackproxied packets of type
fr_packet_list_t• alloc_id is type int, so
proxy_list can track 65Kpackets• The error messages indicate thatthe proxies are exhausting this IDspace/**Structure defining a list of packets(incoming or outgoing)*that should be managed.
*/

struct fr_packet_list_t {rbtree_t*tree;
int
alloc_id;
uint32_t
num_outgoing;
int
last_recv;
int
num_sockets;
fr_packet_socket_t sockets[MAX_SOCKETS];};Allocation errors on primary proxy containers• 13th February 2023
• 1-minute sampling• Typical pattern with
proxy exhaustion
between 1600Z and2000Z• Container 2 is barely
effected – is traffic
volume a factor?

Allocation errors on primary proxy containers
Incident on 13th February• November 2022
through March 2023
• Overall traffic
volumes have notincreased• But proxy allocation
failures haveincreased
significantly sinceJanuaryImpact of international peerings
Transfer of some peerings between containers

Incident on
13th February• The graphs show traffic
data for February as theseverity and frequency ofevents increased• The data gives traffic
volumes for the US
primary’s containers’
international peerings• Allocation errors (blue)
follow the international
peerings

Early observations• Loss of 20-25% of requests for 3-4 hours per day• These episodes increase in frequency and severity from January, becomingnearly daily by mid-February• These episodes of exhaustion coincide with peak usage, at around 1700Z• This is no material increase in load over this period• The episodes appear to correlate with a container’s international peerings

A clue from an Identity Provider• An Identity Provider
raised a ticket
concerning
unexplained
authentication
failures on 16thFebruary• The issue appeared
unrelated initially,
but the “burstiness”
chimed with theepisodic nature of
the first issue

Malformed EAP messages in the proxy logs
• Looking at our RADIUS proxy logs, we see that
• The EAP packet length claimed in the packet’s header does not match thepacket’s actual length• The RADIUS server never responds to the proxied request• The EAP type (“AirFortress-EAP”) has never been implemented• This user has always authenticated previously using EAP-TLS

Analysis of EAP types observed by the proxies• AirFortress is the fourth most widely requested authentication Type – but has never beenimplemented• Types of None and 255 (none and all bits set, respectively) are fifth and eighth – they are bothinvalid values• All Type values (0-255) can be observed within 3 weeks• These unusual values are observed from 755 service providers globally returning Operator-Name
LOG(COUNT_EAPTYPE)
8
7
6
5
4
32
1

3465

1529

231

194

191

162

158

152

151

147

133

132

132

128

126

142

4662

146

6557

14356

166615TLS Identity TTLS

1334795

4412411

PEAP3725717

18739304

0

NAK
AirFortress-EAP
None

PWD

FAST

255

97

108

112

OTP

100

111

GPSK

254KEA-Validate
MSCHAPv2LEAP

99

70

117

54

Malformed EAP packets are emitted globallyPrevalence of AirFortress/None/255 EAP Types observed on US proxies by realm
500• the presence of the OperatorName attribute

450400• a user from the US visiting aninstitution

350
300

• the malformed packet having a
Type of AirFortress, None, or 255

250
200

However, it demonstrates the global
distribution of malformed EAPemissions

150
100
50
0

ac.at
ac.id
ac.il
ac.in
ac.jp
ac.lk
ac.nz
ac.th
ac.uk
ac.za
at
br
ca
com
cz
dedk
edu
edu.bredu.cnedu.myedu.pkedu.pledu.tredu.tw
eseu
fifr
gov
gr
hr
ie
isitjplunetnlnoorgptuk

Number of Service Providers

This data is indicative because it relies
onRealm

The data is heavily weighted to .edu
because our proxies know theirOperator-Names

EAP Frame (Response/Identity)
The typical EAP-Request/Identity length is around 28 octets or 224 bits
Code: identifies the type of packet (request, response, success, failure) (2)Identifier: matches requests and responses (variable)
Length: length of the EAP packet (variable)
Type: the EAP-Identity value (1)The reported F-TICKS value is an 8-bit windowinto the EAP packet (or < 4% of the packet)

Type-Data: the EAP Identity value (variable, takes theformat username@realm; 23 octets is the average)Three hypotheses• Something is going wrong between the supplicant and the RADIUS client
1. It is a buggy supplicant• This seems the most likely hypothesis: supplicants have been a source of problems in the past
• However, widespread use of MAC anonymisation (~87%) makes it impossible to correlatemalformed EAP packets with supplicant platforms

2.
3.

It is wireless corruption• Wireless corruption is common, but CRC error detection should prevent leakage “onto the wire”• This is no easy way of testing this hypothesis

It is a buggy RADIUS client• This seems unlikely because• vendor products tend to be reliable• it is a bizarre failure for a “pass-through” authenticator that is meant to be transparent at the EAP layer• a significant proportion of SPs are emitting malformed EAP packets and it seems improbable that they (andtheir vendor) are all running buggy products without realising• There was no obvious way of fingerprinting products from the data available in our logs

Three lucky breaks• CSI value format strongly suggests the RADIUS client is at fault• I noticed that about 95% of CSI values associated with malformed EAPpackets use the same EUI-48 format (lower case and hyphen delimited)• The general prevalence of this format is only 59%• This suggests that the malformed EAP packets are associated with theRADIUS client, because that is the entity that creates the CSI value• Capture of a malformed EAP packet• Margaret Cullen and Alan DeKok manage to identify a rare instance of amalformed packet manually using tcpdump• The contents of the packet are weird but intelligible (a DNS message),ruling out wireless corruption• Discovery of the tshark tool

• Provides much more powerful filtering than tcpdump• sudo tshark -w - udp and dst port 1812 and dst host 163.253.31.2 |tshark -V -n -r - "eap.code > 6"

Identifying the RADIUS client vendor• We used tshark to obtain hundreds of malformed EAP packets
for the top emitters• The RADIUS Access-Requests for these packets all includedVSAs for the same vendor• The payloads in the packets were usually unintelligible, but
often recognisable, and sometimes very unusual• The malformed EAP contents are probably parts of random memoryin the access point and/or controller• The example below shows the EAP message attribute (highlighted)
partially taking the value of another RADIUS attribute

The AirFortress-EAP mystery solved

The fifth byte gives the EAP Typeas 25 (hex), which in decimal is 37:
the value for AirFortress-EAP

Managing the issue• The frequency and severity of the disruption continued to increase
through February and March• We increased proxy capacity as a workaround (and so increase the IDspace) by adding additional RADIUS proxy containers• The Internet2 eduroam Ops team discussed the issue with Alan
DeKok at IETF 116 in late March, who provided a software solution• Packets with an invalid EAP code (>6) get an Access-Reject and are notproxied• Packets with incorrect EAP length get an Access-Reject and are not proxied• There is an ongoing discussion on the IETF RADEXT mailing list
concerning the best approachAllocation errors and proxy capacityNew containers created• The graphs show allocationerrors as a percentage of all
requests from the start ofFebruary until end of May• The additional proxy capacity
immediately reduced thefrequency of allocation errors• This temporarily increased thecost of AWS ECS by 24%• Alan’s patches are going into
production very soon
• We expect to remove theadditional containers at somepoint

“Dark EAP packets” on eduroam?
• The EAP Type reported in the proxy logs provides a very limited view (just 8-bits)of the EAP packets that we proxy and so we can only detect a subset of
malformed packets• There may be a larger volume of “dark EAP packets” being proxied that we cannot
detect with our existing instrumentation• We can count the RADIUS packets transporting dark EAP packets, but do weunderstand their impact on the infrastructure?

February 13th, 2023

Some questions remain• Why did the severity of the problem increase after January 2023
when volumes of malformed EAP packets appear to remain static?
• Why do malformed EAP packets from our international peers appear
to have a greater impact than US-sourced packets?
• CSI format analysis suggests there is at least one other vendor
emitting malformed EAP packets – who are they, and do we care?
• We have completed the analysis needed to identify the root cause
and find a solution and so the impetus to investigate these open
questions has receded – but that doesn’t mean they’re not important

Recommendation and discussion points• NROs should consider the implications of this issue• Many SPs globally are emitting malformed EAP packets• Correctly behaving, non-responding IDPs are degrading RADIUS proxy performance• Consider what steps might be appropriate to manage this issue nationally• We are trusting of our RADIUS clients and supplicants• Our access points form a massive, accessible surface for EAP-Requests, benign and malicious• The RADIUS client vendor does not appear to be prioritising the problem• The vendor’s products have had this issue since at least 2018 – does our infrastructure need more
intelligence and resilience to identify and manage future problems?

• We pay much less attention to EAP than RADIUS• But the RADIUS infrastructure exists for the sole purpose of transporting EAP• Should our proxies be applying policy on EAP packets?
• See recent discussion on the IETF radext mailing list – some interesting architectural points