SharkFest ’18 US

Point and shoot packet
Shooting point of field in packet analysis

Supplemental trace files are
http://www.ikeriri.ne.jp/sharkfest/

Megumi Takeshita
Packet Otaku
ikeriri network service#sf18us#sf18us •• Computer
Computer History
History Museum,Museum, Mountain
Mountain View,View, CACA •• June
June 25-28
25-28

Megumi Takeshita, ikeriri network service•••••

Founder, ikeriri network service co.,ltd
Wrote 10+ books about Wireshark
Reseller of Riverbed Technology ( formerCACE technologies ) in Japan
Attending all Sharkfest
Translator of QT Wireshark into Japanese#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Point and shoot packet
When you debug, troubleshoot, and inspectsecurity issues using Wireshark, you may justlook a glance of trace file, and watch eachpacket sequentially in detail.
In this session Megumi show you good ways topoint and shoot packet, using display filters,
graphs and tables. Each layer’s header has
important fields to analyze trace file. So you
know shooting point of field in trace file.
This session show you alternative focus pointsof packet for your debugging, troubleshooting,and inspection.
#sf18us • Computer History Museum, Mountain View, CA • June 25-28

3

Point and Shoot packet• There are trace files for debugging andtroubleshooting, but what, how do we resolve ?
• Now show you the way to point the key in trace
files, then shoot the trouble using Wireshark• Note all trace files are modified and anonymized#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Collect 2 trace files at least!

• If you inspect a trace file sodeeply, you may not find the key.
• Debugging and troubleshooting
are kinds of black box test, sowe should point from outside ofthe system using trace files.
• At least 2 trace files on different
conditions are needed for starting#sf18us • Computer History Museum, Mountain View, CA • June 25-28

CASE1 CATV Box• Customer’s CATV boxes in the hotel• Some boxes fails to start up• Capture 2 trace files
1_DHCP_SUCCESS.pcap
1_DHCP_FAIL.pcap• Let’s start debugging#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Open 2 trace files

Recommendation: create Flow Graph of both

#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Where is the point ?
• DHCP is upgrade version of BOOTP by Microsoft,and RFC defines rough procedures.
(Discover-Offer-Request-ACK)• CATV box, router and small IoT devices uses smaller
and incomplete protocol stack.
• Where is the point ? Destination address frompacket list ? Umm please look Info and open eachpacket dissector of Bootstrap protocol.
#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Transaction ID• Transaction ID is the random number choosed byclient. And we use the same transaction id in thespecific DHCP process.
• Choose Bootstrap Protocol > Transaction ID andright click to Apply as Column• Check both trace
files#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Shoot the Bug
Each DHCP
transactionuses the sameTransaction ID

DHCP Server
returns different
Transaction ID
that CATV box sent

• DHCP server returns different transaction ID that
CATV box sent. The problem is the DHCP server( broadband router ) behavior is not appropriate.
• IoT devices uses poor dhcp software implementation#sf18us • Computer History Museum, Mountain View, CA • June 25-28

CASE2 Cannot see homepage• A OSAKA user complains about the troublesome webpage cannot be displayed ( the others can ).
for example, Google OK but Apple NG.
• TOKYO user says he never experienced
such trouble in the same environments• We capture packets in Osaka(2_OSAKA_FAIL_LAN.pcap) and Tokyo( 2_TOKYO_SUCCESS_LAN.pcap)#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Open 2 trace files

Recommendation: create TCP Flow Graph of both

#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Where is the point ?
• TCP connection starts with 3 way handshake andexchange parameters (MSS, SACK, Scaling etc.)
in first two packet aka TCP negotiation .
OSAKA(FAIL)TOKYO(SUCCESS)
ClientClient

MSS=1460 WinScale=4(16) SACK
MSS=1414 WinScale=512 SACKServerMSS=1460 WinScale=4(16) SACK
MSS=1460 WinScale=512SACK• Tokyo’s Web Server uses MSS=1414, Osaka 1460
#sf18us • Computer History Museum, Mountain View, CA • June 25-28

ServerMaximum Segment Size
• MSS(Maximum Segment Size) is calculated
MSS = IP MTU – IP Header (20) – TCP Header (20)
DatalinkHeader

IP MTU( Maximum Transfer Unit ) Size

IP Header

TCP Header

MSS ( Maximum Segment Size)each IP and TCP header size without Options is 20• But why Tokyo is OK
and OSAKA is NG ?

#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Tapping at both LAN and WAN• To resolve internet connection
problem, we need to capture atboth boundary point of network.
(LAN and WAN side)• We need router side packets.
• Please add to open additional
2 trace files at the WAN side
2_TOKYO_SUCCESS_WAN.pcap
2_OSAKA_FAIL_WAN.pcap#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Open additional 2 trace files
Recommendation: look at TCP negotiation carefully#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Where is the point ?
• Both routers in Tokyo and Osaka rewrite Ethernet
and IP header (ex. checksum), adds PPPoE and PPP
header to connect to access point of the ISP (DSLAM（Digital Subscriber Line Access Multiplexer))• Osaka router does not adjust MTU and MSS size,just route the IP datagram (including TCP segment)
TCP SYN
MSS=1460
Scaling 2 (4)SACKTOKYOrouter

TCP SYN
MSS=1414
Scaling 2 (4)SACK

TCP SYN
MSS=1460
Scaling 2 (4)SACKOSAKArouter#sf18us • Computer History Museum, Mountain View, CA • June 25-28

TCP SYN
MSS=1460
Scaling 2 (4)SACK

The Point is in WAN side
#1 in TOKYO LAN
Original Client SYN#1 in TOKYO WAN
Router rewrites SYN#sf18us • Computer History Museum, Mountain View, CA • June 25-28

MTU/MSS problem• UDP/IP application uses MTU to fragment eachdatagrams, and TCP uses MSS to split eachsegments. TCP sets MSS value in the negotiation.
• MTU and MSS problems are common in Internet
connection using PPPoE and PPP datalink.
• We need to modify router (or host) parameters tomatch ISP’s requirements.
• Some application automatically detect and adjust
this problem. ( ex. PMTU / NDP, and IPv6 !)#sf18us • Computer History Museum, Mountain View, CA • June 25-28

CASE3 Slow Single Sign On• Many Web based application by 1500 users inenterprise system, each application need to login.
• The customer installed SSO server based on reverse
proxy ( HTTP proxy authenticates user’s application )• It took 15 seconds to login from remote, but it takes
about 100 seconds after SSO, users are angry.
LoginScreen

ChooseApplication

Login
Screen(each apps)#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Application

Network diagram
15 seconds to app’s login screen

After Single Sign OnBeforeInternet

100 seconds to app’s login screen

FireWallDMZServerServersFireWallInternetServerServers

SSO Server(Reverse Proxy type)

LANApplication1

Application2…

ApplicationN

Need to manually and interactive
login into each application

DMZLAN
Application1

Application2…

ApplicationN

Login at SSO Server once, then you
do not need to login into applications !

#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Capture at user point first• To check repeatability, we need to capture at user-side point

first (sometimes the issue is in the user specific environments,
or just user’s angry because of another reason. )• Filter and remove other packet for minimizing and simplifying
the trouble, use 2 ( Before SSO / After SSO ) trace files.
(3_Before.pcap and 3_After.pcap)
NOTE: sorry for inconvenience, all payload data is set to zerousing pktanon for security reason• If you cannot find the key, then we need to check the serverside point, so we can save times and money#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Open 2 trace files

Recommendation: create TCP Flow Graph of both

#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Visualizing TCP Data stream• To understand TCP activities, creating theTCP stream graph is a good idea, but there
are many TCP streams in each trace file.
• Using tcp.analysis.bytes_in_flight to visualize
the TCP by I/O graph ( also check TCP pref.)• Set Y Axis to SUM ( Y Field ) to set Y Field as
tcp.analysis.bytes_in_flight• TCP activity is easy to understand with 1ms
interval in common internet infrastructure(Test ping to calculate latency at WAN side)#sf18us • Computer History Museum, Mountain View, CA • June 25-28

TCP analysis bytes_in_flight

About 14 seconds
About 105 seconds in same procedure

You can visualize the latency and TCP activities, but Why ?
#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Capture File Properties
Check the Elapsed Time and Bytes

Time, Packets and Bytes are not so different#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Where is the point ?
Look into latency of each TCP connection• Choose Statistics > Conversations and select TCP tab,Look into latency of each TCP connection.
• Sort ascending by Rel Start and check the grey line,
the band means the time of each TCP stream like
Network tab in Chrome Developer Tool ( like Wireshark 3 !)#sf18us • Computer History Museum, Mountain View, CA • June 25-28

TCP conversation (sort by Rel Start)
Check the grey band and understand each TCP stream#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Each duration of TCP Stream• Sorting descending by Duration Row, you can also
understand the difference of TCP stream.
• Almost over 10 times slower than Before SSO.
• It is the key of the performance problem.

#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Shoot the trouble• Increasing Round Trip Time is one of the reasons.
• Sorry All HTTP/HTTPS payloads are anonymized,
there is a key in reverse proxy server settings.
• Reverse Proxy SSO server never uses cache,
Cache-Control: no-store, no-cache, must-revalidate,post-check=0, pre-check=0, max-age=0, private,
no-transform, proxy-revalidate and Expires: -1• We need to change reverse proxy server
configuration to modify cache settings of SSO server.
#sf18us • Computer History Museum, Mountain View, CA • June 25-28

CASE 4 Smartphone• Only one smartphone fails to connect Wi-Fi, thoughothers can access the network using WPA2-PSK.
• The trouble happens in just onely one client in theentire network, it is a hint to minimize the problem.
• Capture and filter using Smartphone
MAC address (22-22-22-22-22-22)
4_WiFi_Fail.pcap (anonymized trace file)#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Open trace file 4_WiFi_FAIL
This time please think of layer 2 header#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Association Response• We need to determine layer 2 or upper.
Association Response (wlan.fc.type_subtype==1)
is a good indicator of Wi-Fi troubleshooting.
• If you find the Association Response, Datalinkprocedures are almost completed, exchanging
Beacon, Probe Request, Probe Response,Authentication x 2, Association Request then link up
with Association Response packet• Let’s filter packets with “wlan.fc.type_subtype==1”#sf18us • Computer History Museum, Mountain View, CA • June 25-28

EAPOL 4 way handshake• You can find Association Response 3 times ? Why• EAPOL 4 Way handshakes are required in a ordinal
WPA-PSK process. Try to filter with “eapol”#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Where is the point ?
Stack point has a problem to shoot the trouble• Finding stack point is important especially in Wirelesstroubleshooting, in this case, every time the processes
are stopped at Association Response.
• Look into Association Response
carefully to find the reason#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Invalid AKMP

(Adaptive Key Management Protocol)

IEEE802.1x authentication method• Invalid AKMP means key exchange settings aremismatch between Access point and Station.
• This time AP use PSK( Pre Shared Key ) Passphrase,
but STA use IEEE802.1x authentication server.
#sf18us • Computer History Museum, Mountain View, CA • June 25-28

One more
BREAKING NEWS• You may know, Metageek announced Eye P.A.
support capturing IEEE802.11n and then
IEEE802.11ac in Windows environment !
• After the end of AirpcapNX era, wireless packet
analysis on Windows is limited. ( AcrylicWifi and other
NDIS based driver never behaves as we wish… )#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Eye P.A. (Eye Packet Analyzer)• You need Beta version of Eye P.A.
now EyeP.A. supports 11n with v1.12
then 11AC support comes at last week !!
• Start up EyePA with Administrator privilege.
Install NDIS Driver• Then you can capture via
CAPTURE DEVICE CONTROL• Off course you can save andsend trace to Wireshark !
#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Compatible Adapters (1)• Eye P.A. ver1.12 (May 2018) supports 11N
Linksys AE2500, Linksys AE1200, Netgear A6200(NDIS based driver)Riverbed AirPcapNX, AirPcapTX(AirPcap based driver)• Tarlogic NDIS Monitor Driverand AirPcap driver are used
when we use 11n in ver1.12#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Compatible Adapters (2)• Eye P.A. Version 1.13 (A.k.a Imperial Eye P.A.)• Support adapter:
ASUS USB-AC56, ASUS USB-AC68, ALFA Network
AWUS1900, Linksys WUSB6300, Amped Wireless
ACA1, EnGenius EUB1200AC, D-Link DWA-182 rev
C1, D-Link DWA-192, TRENDnet TEW-805UB, TPLINK Archer T4U v2, TP-LINK Archer T4UH v2,Edimax EW-7822UAC, Edimax EW-7833UAC#sf18us • Computer History Museum, Mountain View, CA • June 25-28

TEST: Linksys AE2500• Linksys AE2500 Dual Band N
USB3 2.4/5GHz USB2
BCM4323 (Broadcom chip)• CH 1-14,CH 36-64 W52,
CH 100-165 W52 W53 in Japan• Automatically recognized in Eye
P.A. version 1.13.0.13• You can choose Packet Data istruncated or not#sf18us • Computer History Museum, Mountain View, CA • June 25-28

TEST: Linksys AE2500#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Compatible Adapters (3)••Eye P.A. ver2 supports 11AC•

802.11n adapters
Linksys AE2500 (recommended), Linksys AE1200, Netgear A6200
NOTE LinksysAE1200 and NetgearA6200 cannot be used on DFSCH••RiverBed AirPcap NX, AirPcapTX

Supported adapters (IEEE802.11ac)
Linksys WUSB6300 (recommended), ASUS USB-AC56, ASUS USBAC68, ALFA Network AWUS1900, Amped Wireless ACA1, EnGenius
EUB1200AC, D-Link DWA-182 rev C1, D-Link DWA-192, TRENDnet TEW805UB, TP-LINK Archer T4U v2, TP-LINK Archer T4UH v2, Edimax EW7822UAC, Edimax EW-7833UAC

Tamosoft ( famous as TamoGraph) driver is used.
#sf18us • Computer History Museum, Mountain View, CA • June 25-28

TEST2 : ALFA AWUS036ACH• Alfa Long-Range Dual-Band
AC1200 Wireless USB 3.0 Wi-Fi
Adapter w/2x 5dBi External
Antennas – 2.4GHz
300Mbps/5GHz 867Mbps –802.11ac & A, B, G, N
• RTL8812AU Realtek chipset• CH 1-14,CH 36-64 W52,
CH 100-165 W52 W53 in Japan#sf18us • Computer History Museum, Mountain View, CA • June 25-28

TEST2 : ALFA AWUS036ACH• This ALFA model is notofficially supported, but it
try to install specific drivers• Automatically recognized in
Eye P.A. version 1.13.0.13• You can choose Packet
Data is truncated or not#sf18us • Computer History Museum, Mountain View, CA • June 25-28

DEMO : ALFA AWUS036ACH#sf18us • Computer History Museum, Mountain View, CA • June 25-28

Thank you for attending !
どうもありがとうございました#sf18us • Computer History Museum, Mountain View, CA • June 25-28