**An Overview of ICMP Redirect Traffic Issue at NERSC**

**Scott Campbell**

**Stephen Lau**

**NERSC**

**November 14, 2023**

1.  **Overview**

During the testing period of the Snort intrusion detection system at
NERSC, it became apparent that there were a large number of ICMP
Redirect packets crossing NERSC\'s network border which were
non-standard in their profile. Initial analysis of the data and
investigation of data recorded by our intrusion detection system
indicate that the packets have been arriving for longer than the trace
records are archived. Due to the unique nature of the traffic, detailed
analysis was started. This report summarizes what we have uncovered so
far in the investigation. This report is not a conclusive summary, just
an informal report of our current findings.

2.  **Introduction to ICMP Redirect Traffic**

ICMP traffic is, in general, used for inter-communication and control of
the Internet infrastructure. It is not normally used for direct client
to client/server communication for day to day traffic.

The function of an ICMP Redirect packet is as follows. There is only one
scenario when an ICMP Redirect packet should be sent. If during transit,
a gateway receives a packet bound for a remote network, and that gateway
determines that the next hop to get to that remote network is on the
same network segment as the source host of the packet, a Redirect
message is sent to the host. This Redirect tells the host that it should
direct any future traffic to the remote network in question directly to
the next hop gateway, rather than the one sending the Redirect.

How this works is as follows (Stevens, p.119):

![](media/image1.png){width="4.21875in" height="2.3854166666666665in"}

(1) We assume that the host sends an IP datagram to Rl. This routing
    decision is often made because Rl is the default router for the
    host.

(2) Rl receives the datagram and performs a lookup in its routing table
    and determines that R2 is the correct next-hop router to send the
    datagram to. When it sends the datagram to R2, Rl detects that it is
    sending it out the same interface on which the datagram arrived (the
    LAN to which the host and the two routers are attached). This is the
    clue to a router that a redirect can be sent to the original sender.

(3) Rl sends an ICMP Redirect to the host, telling it to send future
    datagrams to that destination to R2, instead of Rl.

These redirects are not normally seen on the Internet side of border
routers, and have historically been considered a security problem based
on their ability to adjust a remote host's routing table. This
capability can be used in a wide number of remote attacks.

A normal ICMP Redirect packet looks like the following:

![](media/image2.png){width="5.3125in" height="1.9479166666666667in"}

What is important to see here is two things. The first is the size of
the original IP datagram data. In a \'real\' redirect packet, we see
that you get the header and 8 bytes of the original data. In the packets
that we were seeing, the entire original packet was included in the
redirect payload. The second thing to notice here is the expected
distribution of the IP addresses. When you take the packet apart, you
have six addresses to look at. They are:

  ----------------------------------- -----------------------------------
  Name                                Notes

  IP Source                           

  IP Destination                      Same as data source

  ICMP redirect destination           Same as data destination

  ICMP new gateway                    

  ICMP data source                    Same as IP destination

  ICMP data destination               Same as ICMP redirect destination
  ----------------------------------- -----------------------------------

From this it can be seen that we can expect considerable overlap between
several fields within each of the packets. This is in fact seen in the
data crossing the border.

3.  **Introduction to the data and its unique characteristics**

There are several things that make the ICMP Redirect packets seen
flowing across the NERSC network border interesting. The first, already
mentioned, is the existence of a full packet of data within the redirect
itself. This is what initially set off the intrusion detection system in
the first place. After doing an extensive search on the Internet, and
discreetly asking several network engineers, it became apparent that not
only was this unusual behavior, but nobody has ever seen this sort of
behavior before nor could they come up with a plausible explanation for
its occurrence.

Another unusual aspect is the number of redirects seen. Removing the
suspect data, we see only a handful of these packets crossing the border
over a period of 60 days. The mere existence of a redirect packet on the
border router is reason for being suspicious, and usually involves a
routine security analysis of the packets. In this case we are seeing
20,000-30,000 packets per day, with a time distribution that correlates
with an external business day offset by 8-9 hours from local Pacific
Time (more below).

The final thing to mention is the address distribution that presents
itself with the packets. If you have a single network and a pair of
routers (like in the example above), even if you see thousands of
packets from many hosts within the internal network, the distribution of
addresses within these packets should start regular and become more and
more predictable over time. In the data collected we are seeing hundreds
of addresses smeared across large areas of address space. This behavior
is most unusual.

These represent the most basic irregularities of the data. There are
several more, but to simply express the situation, the data makes little
sense to anyone who has seen it.

4.  **Data Analysis**

The following represent a number of tests and graphs which explore the
distribution of various aspects of the data. There are basically two
groups of data. The first is the ICMP redirect packets themselves, and
the second is the data within the packets. Since the redirect packets
contain complete IP packets within them, we were able to extract and
analyze this data as regular (complete) Internet traffic.

A breakdown of the analysis follows.

1.  **ICMP Packets**

Each following section will detail information extracted form the ICMP
redirect traffic.

**4.1.1 Time distribution of source IP host and source IP net as % of
daily traffic**

Given such a large number of source IP addresses, it is reasonable to
wonder what the daily/time distribution of the address space is. For
example, if a small number of addresses (A) are represented every day
with a large percent of the daily total traffic, while a large number of
address ranges (B) are randomly cycled through, one might suspect that B
is being used to hide the traffic found in A.

While the initial analysis focused on host IP addresses, it became
evident that there was too much data to analyze in a useful manner. We
subsequently reduced our analysis to looking at network data.

Looking at the basic data graph, we see a number of interesting things.
Keep in mind that the tools (and hence the graphs) used are not perfect,
but provide high speed analysis and are extremely flexible in their
abilities.

Here we are looking at a number of the regular contributors:

![](media/image3.png){width="6.0in" height="4.5in"}

What we are seeing here is the percentage of total traffic that each
network represents each day. The \"Network Subnet\" value is just an
indexing value and can be ignored. The subnets are color coded with a
key in the upper right corner of the graph with the far left graph
correlating to the top entry in the list. The important things to
observe here are the regular (large) percent of traffic and the cyclical
consistency of the client IP addresses. The cyclical depression which
aligns itself across all of these examples seems to be weekly in nature.
To contrast this consistent behavior, a collection of the remaining
subnets is presented.

![](media/image4.png){width="6.0in" height="4.5in"}

Looking at this data, not only are the total percents smaller, but there
is far less overlap than in the other set along the time axis. The
implications of this are that the IP addresses of the consistent sources
might be masked behind the noise represented here. Unfortunately this
simple explanation fails to explain why there seems to be good data
contained within these sparse subnet fields, and could probably be
explained away by some mysterious misconfiguration error. More
investigation relating the various IP addresses is in order to see if
they can be related in some predictable way. An example of this in the
past has been a broken TCP stack which improperly translated byte
orders, making seemingly mysterious \'random\' data quite reproducible.

A graph with the total data collection is reproduced in the appendix for
reference.

**4.1.2 Number of individual networks represented per day**

An interesting aspect of the data is the distribution of the total
number of discrete IP source addresses in the ICMP redirect packet.
While the basic distribution is not all that strange, if it is taken in
light of the information in 4.1.1 the numbers seem a little unusual.

![](media/image5.png){width="6.0in" height="4.5in"}

The nice symmetric weekly pattern stands out clearly. Most of the
asymmetry between the weeks can be accounted for by missing data from
the sample sets. What is most interesting is that if the source IP
addresses were random in nature, we would expect there to be less
symmetry in the distribution. Again, the pattern here is strange, but
does not hold any sort of real clue as to the nature of this traffic.

**4.1.3 List of source, destination, ICMP redirect destination and ICMP
redirect gateway in total**

In order to reduce the effects of data loss and short term patterns
which are not relevant to the overall distribution, a series of tables
were prepared which display the source, destination, ICMP redirect for
and the ICMP redirect gateway IP numbers. The tables themselves will not
be directly presented here, but will be inserted into the appendix for
your analysis. One thing to note is that we did not remove any of the
data, so that there will be a small number of hosts listed which are not
directly related to the situation being looked at here. The reason for
this is that it not always immediately apparent what is real data and
what constitutes \'regular\' cross border ICMP redirect traffic.

There are several things interesting about each grouping of the data.
This will be detailed in each of the following sections.

**4.1.3.1 ICMP Source IP Address**

There are many interesting aspects to the source IP collection, but by
far the most informative is traceroute information. A traceroute shows
the path that packets take across the Internet. Using this tool, it is
possible to see if two IP numbers are \"near\" one another.

Looking at the traffic in the same two groups (regular vs. sparse), we
see a number of interesting things. A traceroute from one of the regular
networks (203.240.222.0/24, 203.240.225.0/24, 211.191.225.0/24) looks
more or less exactly like:

traceroute to 203.240.222.228 (203.240.222.228), 30 hops max, 38 byte
packets

1 adsl-63-199-31-254.dsl.snfc21.pacbell.net (63.199.31.254) 14.956 ms
14.091 ms 12.689 ms

2 dist3-vlan50.snfc21.pbi.net (206.171.134.132) 15.162 ms 14.397 ms
12.840 ms

3 bb1-g1-0.snfc21.pbi.net (209.232.130.28) 12.049 ms 12.871 ms 13.165 ms

4 bb2-p12-0.snfc21.pbi.net (64.161.124.50) 14.987 ms 14.830 ms 13.175 ms

5 206.223.120.85 (206.223.120.85) 14.448 ms 16.369 ms 16.358 ms

6 210.180.97.116 (210.180.97.116) 16.205 ms 17.566 ms 16.039 ms

7 210.180.97.213 (210.180.97.213) 142.011 ms 210.180.97.1 (210.180.97.1)
144.674 ms 145.037 ms

8 211.37.96.2 (211.37.96.2) 142.301 ms 141.123 ms 143.112 ms

9 211.117.40.62 (211.117.40.62) 139.793 ms 148.443 ms 138.229 ms

10 202.30.243.251 (202.30.243.251) 163.218 ms 145.265 ms 164.786 ms

11 changhyun-4500-2-s.shinbiro-seoul.route.shinbiro.com
(211.191.255.226) 174.406 ms 174.748 ms \*

12 10.89.255.11 (10.89.255.11) 208.045 ms 170.180 ms 203.384 ms

13 \* \* \*

The main difference between the subnets is the exact host number on the
10.89.255 network. Sometimes it is 11, sometimes 12. The most
significant thing about this is that the 10.0.0.0/8 network should not
be seen on the \'live\' Internet since it is a private address space.
The 211.191.203.0/24 and 211.191.204.0/24 subnets also share an
interesting host - the last real IP number before the 10.89.255.11. This
may be indicative of the actual address space that we are looking for.

The remaining address collections seem to share in a number of identical
hosts near the final destination. There are two general address groups
which can be represented by the 211.227.0.0/16 and 61.81.0.0/16
networks. Sample traceroutes to 211.227.228.107 and 61.84.185.114, which
would seem like different addresses, share the same next to last hop
which places them almost next to one another in a network sense.

traceroute to 211.227.228.107 (211.227.228.107), 30 hops max, 38 byte
packets

1 adsl-63-199-31-254.dsl.snfc21.pacbell.net (63.199.31.254) 14.564 ms
13.623 ms 12.846 ms

2 dist3-vlan50.snfc21.pbi.net (206.171.134.132) 12.062 ms 14.141 ms
13.156 ms

3 bb2-g1-0.snfc21.pbi.net (209.232.130.29) 15.312 ms 14.098 ms 13.162 ms

4 bb1-p11-0.sntc01.pbi.net (64.161.124.245) 16.771 ms 15.422 ms 16.387
ms

5 64.161.1.26 (64.161.1.26) 14.718 ms 15.473 ms 13.160 ms

6 166-49-238-205.concert.net (166.49.238.205) 17.277 ms 17.542 ms 16.639
ms

7 t1c2-ge6-0.us-sfo.concert.net (166.49.228.2) 18.529 ms 17.506 ms
19.285 ms

8 t1c2-p9-0.us-la.concert.net (166.49.240.74) 24.877 ms 27.164 ms 26.173
ms

9 t1a1-ge8-0-0.us-la.concert.net (166.49.227.39) 25.119 ms 25.803 ms
26.212 ms

10 166-49-252-22.concert.net (166.49.252.22) 24.916 ms 27.205 ms 26.706
ms

11 glgate194-p10-2.kornet.net (211.48.63.9) 183.844 ms 179.037 ms
178.637 ms

12 218.145.63.137 (218.145.63.137) 208.639 ms 207.916 ms 208.868 ms

13 211.193.39.68 (211.193.39.68) 176.247 ms 177.979 ms 178.370 ms

14 211.196.155.126 (211.196.155.126) 182.082 ms 183.212 ms 181.004 ms

15 kj-r1-ge0.kornet.net (203.232.120.9) 213.355 ms 217.072 ms 210.936 ms

16 211.194.52.78 (211.194.52.78) 211.110 ms 213.684 ms 212.180 ms

17 \* 211.198.216.37 (211.198.216.37) 215.453 ms 213.968 ms

18 211.227.228.107 (211.227.228.107) 229.205 ms 227.535 ms 229.765 ms

and:

traceroute to 61.84.185.114 (61.84.185.114), 30 hops max, 38 byte
packets

1 adsl-63-199-31-254.dsl.snfc21.pacbell.net (63.199.31.254) 11.833 ms
13.885 ms 13.165 ms

2 dist3-vlan50.snfc21.pbi.net (206.171.134.132) 15.472 ms 14.166 ms
13.176 ms

3 bb2-g8-0.snfc21.pbi.net (209.232.130.83) 15.298 ms 14.455 ms 13.621 ms

4 bb1-p11-0.sntc01.pbi.net (64.161.124.245) 16.015 ms 15.422 ms 16.440
ms

5 bb1-p6-0.pxpaca.sbcglobal.net (64.161.1.18) 16.073 ms 15.922 ms 16.135
ms

6 pos9-2.mpr1.pao1.us.mfnx.net (64.125.31.205) 15.332 ms 15.476 ms
16.613 ms

7 208.184.232.177 (208.184.232.177) 15.845 ms 17.181 ms 16.380 ms

8 pos0-0.er2a.sjc3.us.mfnx.net (208.185.175.190) 15.342 ms 15.834 ms
16.622 ms

9 64.124.78.245.kt.co.kr (64.124.78.245) 15.322 ms 16.463 ms 16.308 ms

10 glgate136-p7-3.kornet.net (211.48.63.93) 207.518 ms 207.011 ms
208.639 ms

11 218.145.63.7 (218.145.63.7) 208.612 ms 210.772 ms 211.762 ms

12 hh-c2-ge6.kornet.net (211.217.32.132) 210.607 ms 208.629 ms 209.106
ms

13 218.145.32.194 (218.145.32.194) 214.814 ms 214.902 ms 215.809 ms

14 211.194.52.78 (211.194.52.78) 213.987 ms 216.140 ms 215.476 ms

15 \* 211.198.216.37 (211.198.216.37) 216.792 ms 217.183 ms

16 61.84.185.114 (61.84.185.114) 229.989 ms 232.954 ms 235.416 ms

There seems to be no connection between 211.198.216.37 and
211.191.255.226.

The differentiation between the same sets of regular vs. space traffic
is quite interesting. It seems to provide a strong indicator that there
are two discrete address groups.

**4.1.3.2 ICMP Destination IP Address**

The destination IP pool provides us with far less in the way of solid
information to work with. This is because it is necessary for the IP
addresses to be within the range of NERSC\'s 128.55.0.0/16 address
space. The only \'interesting\' thing that stands out is the large
number of addresses whose destination sits on an unused address blocks.
Even this might be explained away by arguing that the NERSC address
space is so sparsely used that this is simply a byproduct of the
situation.

**4.1.3.3 ICMP \"Redirect For\" IP Address**

The \"Redirect For\" IP number identifies the address that you are
redirecting for. This is an interesting number in that this ought to
give us a good idea where the source of this traffic is trying to
connect to. This collection of hosts represents some of the most
confusing data we have to look at. A number of the addresses here seem
to have no Internet route (which is not all that unusual). Others (like
128.10.1.121 and 128.11.1.121) are quite alike, yet seem unrelated when
looking at the routing data. There is a block of addresses
(128.15.0.01/6) which seems to be llnl.gov, and there is even a host in
Japan which seems to be stuck in a routing loop. At this point in the
analysis we have not had the time to look into many aspects of this
data.

**4.1.3.4 ICMP \"Redirect To\" IP Address**

The \"Redirect to" addresses indicate where the redirected traffic ought
to be sent. Comparing this address to the destination IP of the redirect
packet, we see that there is a correlation between the two subnets. From
this we may be able to extract possible alternate destinations for other
redirect traffic. With the NERSC addresses removed, we have the
following as other possible destinations.

Count IP address

\-\-\-\-\-\-- \-\-\-\-\-\-\-\-\-\-\-\-\-\--

1 128.1.1.145

242 128.1.1.146

6 128.1.1.210

998 128.10.1.121

265 128.11.1.121

336 128.15.55.22

348 128.2.1.121

2489 128.2.1.122

398 128.2.1.123

124 128.2.1.130

264 128.22.1.121

246 128.33.1.121

484 128.4.1.121

624 128.4.1.122

4922 128.41.1.121

2383 128.61.1.152

6 130.113.103.133

8 192.168.102.15

5 192.168.18.98

2 208.148.46.107

165 okayama-1.sinet.ad.jp

It would be interesting to see if similar redirect traffic can be
found - currently we are working with ESNet to track down this possible
traffic in networks under their control.

**4.1.4 Time distribution of all packets**

The final thing that we looked at was the distribution of all received
packets across a 24 hour period. The distribution of the data looks like
the following.

![](media/image6.png){width="6.0in" height="4.5in"}

What the data shows is the number of connections/second for the sum
total of all the data. The graph should be looked at more for its shape
than anything else since the sorting process removed any of the zero
size entries. What we would like to point out us that the distribution
looks like a workday offset by a little over half a day from local
Pacific Time. This would correlate with a business day in Eastern Asia
or Australia.

2.  **Inner Data**

The inner data was retrieved by writing a custom software application
which stripped off the ICMP redirect header. This provided complete
tcpdump files for the redirect data inside. This is quite useful, except
that correlating events between the two tcpdump files requires manual
alignment, which is quite time intensive. Because of this, much of the
important correlation analysis has not been completed yet.

**4.2.1 Distribution of source and destination inner IP address**

The inner source IP addresses are remarkably uninteresting in that they
correlate exactly to the IP destination addresses. This is what we would
expect.

The destination addresses also correlate with the ICMP redirect
destination. Again, this is exactly what we would expect. When the
protocol analysis is added into the mix, we see that each protocol seems
to have a particularly significant IP number attached to it. In no case
does it seem like there is a clean path to the IP number in question.

**4.2.2 TCP session analysis**

By using the TCP protocol\'s stateful nature, it is possible to track a
TCP connection from beginning to end and have a good idea if there is a
continuous link during the lifetime of the connection. When we do this
for several test connections within the data, we see that the
encapsulated data represents a single sided (client) TCP session,
complete from beginning to end. This is useful in showing that the
distribution of data is at least systematic and more or less whole. This
rules out a \'shotgun\' effect with packets being sent out in a random
manner.

**4.2.3 Encapsulated protocol analysis**

Looking at the distribution of encapsulated protocols provide for some
interesting information. We have not done an analysis for all the
captured data, but looking at each of the inner data files individually
it seems that \~80% of the data is composed of UDP traffic running the
RX protocol. This is running consistently from source port 7000
(afs-fileserver) to destination port 8000. Also seen are source port
7005 (afs-volserver) and 7002 (afs-prserver). The data contained within
the source port 7000 packets seems to decode based on the RFC provided
for the RX protocol.

The RX protocol data contains the majority of the \'interesting\' data.
In there we see what look like typical database query and replies:

CRM_01_CJA_UPDATE

06600050000000208000000000000000000000000660005000000WS02WMK11566020020610

0021001002002061010070070000000000000000000000000000000000000000000000000000000000000004408151057115001

There are hundreds of these transactions in each of the daily files.
Imbedded within the data structures, we have found IP numbers which
correlate to the destination IP of the external packet. This is another
argument against the accidental modification of data along the wire.

The other protocols found include telnet, limited ftp, http and a
handful of unusual protocols. The unusual protocols were most likely
analyzed incorrectly by tcpdump. An interesting find is that http
traffic destined toward certain hosts (all in kjbank.com) is captured,
while all other traffic seems to be excluded. What is equally
interesting is the complete lack of SMTP or POP/IMAP traffic.

All telnet, http, ftp and dns traffic has been extracted into composite
files for further analysis.

**4.2.4 DNS extraction**

DNS data can represent a high content source of information with regard
to network traffic. As expected, there are unusual patterns found in the
data. Three things stand out most.

First is the volume of data. Even for a quiet Microsoft or UNIX based
client, you see a great deal of DNS traffic radiating from it. In this
case we are looking at the combined traffic for 60 days and the total
number of referenced hosts is less than 150. This seems to imply that we
are not seeing the complete traffic on the wire for even a single host.

The second is the \'feel\' of the hosts being looked up. The disposition
of the traffic seems to suggest a single, non-technical user. The
cross-section of hosts runs the range from banking sites (kjbank.com) to
Yahoo! and even a few pornographic web pages. What is interesting is the
large number of resolver queries to the \'daum.net\' domain. This is the
largest ISP in South Korea.

The last significant thing is the presence of reverse name lookups in
the DNS data. This is quite unusual for a client to be conducting this
sort of interaction. One normally expects to see this on the server side
of an interaction. These reverse lookups do not correlate with any
source IP from the ICMP traffic, but seem to come from the same general
net block (they are from the 211.233.79.0/24 subnet).

3.  **Hypothesis**

Creating a model based on what we know about the data has proven to be
remarkably difficult. The more we look into the information we currently
posses, the less clear exactly what is happening. One of the most
difficult thing to work into a model is the address space being used.
There is no good reason why anybody would use the 128.0.0.0/8 subnet for
a personal address space, let alone a legitimate company. In spite of
this, we have developed a possible scenario which will explain at least
a good part of what we are seeing.

4.  **Conclusion**

In it\'s current state it is impossible to determine the exact nature of
the traffic that we are seeing across our border. We simply have not
created a sufficiently detailed model to explain what it is that we are
seeing. It is not the lack of data which is the problem - every night we
get thousands of new packets, but rather problems understanding this
scenario in how it is represented by the data.

We are currently working with ESNet to determine the entry point of this
traffic into ESNet. ESNet is monitoring all of their network peering
points to see if any similar types of traffic are flowing to other ESNet
sites and to determine what external network is the source of this
traffic into ESNet. We are also continuing to analyze this traffic in an
attempt to determine any other patterns that might shed light on its
nature.