PA-3 Debugging Wireless with Wireshark
Including Large Trace Files, AirPcap &Cascade Pilot
Megumi Takeshita, ikeriri network service

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp1Megumi Takeshita, ikeriri network service
a.k.a. packet otaku• Founder, ikeriri network service co.,ltd
since 2002 ← Enterprise solution,
Nortel networks ←Bay Network• Wrote 10+ books about packet
capturing, analysis, inspection, andconsulting ( in Japanese )• Reseller of Riverbed Technology( former CACE technologies ) andMetageek in Japan• Packet capturing Otaku ( geek ) fromEthereal, 1st Sharkfest !
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
2

Ikeriri network service co., ltd.
Packet capture company

Training at JGSDF
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
3

•
••
•
•

Consulting
Reselling
Debugging
Investigating
Training

PacketCapture

Reseller of Riverbed Technology,Metageek, OSCIUM, Dualcomm etc.
• Ikeriri is one of the reseller of Packet capture /analysis products in Japan• Riverbed Technology’s AirPcap, TurboCap, Pilot• Metageek Wi-Spy and Chanalyzer• OSCIUM products• Dualcomm productsetc.
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
4Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
5boundary value analysis andequivalence partitioning• Packet capture debugging is like a Black box-test• Use Pcap/pcapng for boundary value analysis
two ore more pattern / type of the issue
OK pcap and NG pcap,
setON pcap, setOFF pcap
• Collect Pcap in less experiments using equivalence
partitioning ( grouping same environment pattern )We choose only 1 pcap of them and test
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp6Comparison pcap files• We should capture comparison pcap files fordebugging because there may be clues !
• Using boundary value analysis and equivalence
partitioning, collect comparison pcaps.
• Some cases we can easily find the problem, keys,
and the answer only watching 2 pattern of pcaps.
• Frame color,
Expert info is easiest
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp7Gathering information and making table
Create plan ofExperiment• Hearing the customer in deep,address (MAC,IP) port (TCP,UDP)
log message, how to ? How many ?
• Host type, OS, Software version*Android is difficult ( many variation)
iOS ( iPhone and iPad ) is simple
Windows 7/8 may be in same result
Frequency
is also important
Ikeriri•network
service co., ltd
http://www.ikeriri.ne.jp
8

Test capture
procedure

the iteration
numbertest kind, types
test configuration

Standards and protocol and sequence• Standards, protocol helps us
debugging, using documents,
White Papers in IEEE, RFCs
in IANA and other sites• Sequence diagram is verymuch hint for debug forchecking and comparing

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
9Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
10Before capturing• Clear browser cache for capturing all communicationpacket.
• DNS cache is also clear if you need to get DNS queryresponse packet• Disable or turn off Windows firewall and personal
firewall etc.
• Stop and exit software and service of sending packet
like VPN(keep alive), UPnP(SSDP discovery), iTunes• Record Date, IP address, tcp port and MAC address forinspecting later.
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
11Tips1 redirecting information• Executing ipconfig and getmac command andredirecting help us inspecting later
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
12

TIPS2: netstat –a and netstat -b• Show tcp/udp connections using netstat,and I recommend piping and find matching (LISTEN)
netstat –b tells bind application to socket.
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
13

TIPS3• Please check your NIC status ( including Error andDiscard frames ) using netstat –e command.
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
14Check settings in NIC• Today almost NICs offload tcp, udp/ip function.
• Almost NICs support Gigabit Ethernet and carrier
extension ( over 1500MTU ex. 9kb MTU)
• Wireshark read pcap stream from WinPcap• Please check offload settings in properties
in NIC ( from device manager)• Also please check MTU setting too.
(Jumbo frame or MTU)
15

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp

Use Windows Search Index• To add extension of cap and pcap,
set type as clear text search,
We can search pcap/cap files like Google !
off course in multibytes ( in Japanese )• Control panel -> index option / folder option

16

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
17Capturing many interface in one time
Check multiple interface and capture• In case of checking many interface in the same time,
now check multiple interface and start capture.
• Trace file is combined with multiple interface• For example upstream/downstream from router,client/server and so on.

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp18

USB Debugging• We can capture USB frames using Linux• VMware environment also works

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
19Using display filter• Protocol.field.value style• Easiest way is taking use of actual header field(right click and show submenu and set/prepare filter )• Condition of multiple format &&(AND) ||(OR)parameter value can be compared ( gt ge / lt le )• Automatic complication will help you to create• Contains keyword
http.request.url contains ikeriri
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp20Mark and export specified packet
Print packet information to text file
• Marking packet is important for good report.
• Export specified packet and create good trace file.
• Text-based packet information is usable to send
email and making report.

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
21

ＴＩＰＳ：Useful shortcut
Shortcutcontents

Ctrl+↑,↓

Set mouse in packet detail pane, easy to go next / backprevious packet ( useful !! )←,→Expand / collapse information

Ctrl+O,Ctrl+W,Ctrl+P,Ctrl+P,Ctrl+S,Ctrl+Q

Open, Window Close, Print, Save,Quit

Ctrl+H

Output Hex data ( for exporting raw data )Ctrl+F

Find packet

Ctrl+TSet reference time ( for calculating response time)

Ctrl+Shift+P,
Ctrl+Shift+A

preference
ProfileCtrl+[Space]

Immediately clear temporary coloring rules.

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
23Capturing PING(ICMP) packet• Start capturing, then test ping command• Source / destination IP and MAC address is necessary forcommunication under TCP/IP• ARP request / response loop make address resolution.
• ARP result is remembered and cached for 120 seconds ineach PCs• ICMP echo request / response loop check layer 3 connectivity.

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp24

IP trace file analysis• Check identification field of IP header
same Identification number means re-send packet,fragmentation, and security problem.
• TTL field is the hint of hop counts ( always the node
uses 128/64 )• Check DF/MF bit and offset field in IP header.
• Compare IP length field and MTU size.
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
25

ping a.b.c.d –l 1500 -f• original
Ethernet II(14)IP(20)ICMP(8)Message
1500

IP(20)
DF= MF＝offset＝

ICMP(8)Message1472

IP(20)
DF= MF＝offset＝

Message28• Fragment1
Ethernet II(14)• Fragment2 Ethernet II(14)Ikeriri network service co., ltd
http://www.ikeriri.ne.jp26

Count packet size (MTU1500)
ICMP -28
Ethernet II(14)IP(20)ICMP(8)Message1472(MTU=1500)

 ping IP –l size ※-f fragment disabled

TCP HTTP and many protocols -40
Ethernet II(14)IP(20)

TCP(20)

Segment size
MSS＝1460UDP VOIP and video transmission -28
Ethernet II(14)Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
27IP(20)

UDP(8)

Datagram size1472(MTU=1500)

PPPoE Header and MTU size according
to Japanese ISPs
• NTT east flets
MTU 1454Bytes MSS 1414Bytes• NTT west flets premium
MTU 1438Bytes MSS 1398Bytes• GRE + IPsec (transport mode) 1440 Bytes
GRE + IPsec (tunneling mode) 1420 Byte
• UDP(NAT Traversal）
IP(20) UDP(8) , PPPoE, PPP header
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp

(C) いけりり★ネットワークサービスhttp://www.ikeriri.ne.jp29

tfgen• For checking TCP vs. UDP
it is very usefulIkeriri network service co., ltd
http://www.ikeriri.ne.jp30Create IO graph and visualize• Compare TCP ( connection oriented ) and UDP(connectionless protocol ) and visualize.
• Lets use IO graph function and filter packet by protocol• Set X axis to seconds and Y axis to bit/tick (means bps)

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp31Check streams TCP/UDP• Wireshark set stream ID (tcp.stream) in each TCP
connection automatically.
• Filter by tcp stream number and colorize conversation.
• Check bytes using “Follow TCP Stream”• UDP stream is also analyzing by “Follow UDP Stream”

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp32

Export function is very good for HTTP
• We can restore HTTP data from WEB communication
pcap/pcapng files by File>Export>Object>HTTP• HTTP statistics is importantthe count value means Web application performance
1 image map vs. 100 gif file

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp
33

FlowGraph gives you a new look of debugging• Statistics>FlowGraph and maximize the screen• Display filter is very good ways to create good visualization.
• If you need to follow TCP, set graph to TCP graph.
• Compare behavior with RFC and standards

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp34

Trend analysis BASIC
TopN style, and drilled down in details

1. Create TOPN list table of Endpoint and filtered
2. Create N<>other list table of Conversation
3. Then create protocol hierarchy and check stream
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp

35

Utilize IO graph in two ways
• Set packets to Y axis to create ERROR graphHistogram style is good for Frequency graph• Set bit to Y axis to create BPS graphline style is good for amount graph.

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp36

Digest Auth SUCCESS/FAIL

アクセスー認証（失敗）ー認証（成功）.pcapng• Digest authentication will be failed when ID/Password mismatch

401 Unauthorized• If success

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp37Sample trace• Try to click “Home” and check trace file.

• Once called control.cgi and c.1.ae.brightness==0c.1.wb==auto c.1.shade==off c.1.focus==auto c.1.zoom==6040
c.1.pan:=-4014 c.1.tilt:=-153
value send to the server• Moving picture needs 5Mbps
how about creating IO graph and set Y axis
as a bit/tick

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp38

(C) いけりり★ネットワークサービス
http://www.ikeriri.ne.jp
39

Use AirPcap and set clear text if possible• Need Jumbo frame or IEEE802.11a/n go NX• We have to capture their own 4 way handshake todecrypt pcap file secured by WPA2-PSK,• Its terrible troublesome to match between the WPA2
handshake and the communication packet.
• Set free channel
in test capture( android 14ch NG)
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp

1年保守なしType/Subtype, TX rate, BSSID, CH, RSSI• In Wireless environment, please watch importantfield of IEEE802.11 header, physical (radiotap/PPI)
header ( Type/Subtype, TX Rate, BSSID, CH, RSSI)• Many troubles are occurred before Data exchange
hostAP

Prove Req/Ack
Authentication
Association Req/Ack
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp

Data

1年保守なしBetween deployments and standards• IEEE802.11 and related standards, protocols are
not so punctual and irritate rules ( they are notdescribed in detail and all step, procedure, but justset the summary )• For example WPS is famous and many user use
the PIN or button settings, but the deployments in
Wireless devices differs a lot• We have to check sequencesin detail for debugging
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp42(C) いけりり★ネットワークサービス
http://www.ikeriri.ne.jp
43

Huge packet case
• In old days we use sampling technologies
like SNMP, MRTG, and many flow analysis
such as Cisco NetFlow, sFlow, iFlow
IgnoredIgnoredIgnoredMay be• But small packet ( 64 bytes – 100 bytes ) may be ignored.
Some small packet is important symptom of analysis ( ARP/ TCP SYN / HTTP GET and others )Ikeriri network service co., ltd
http://www.ikeriri.ne.jp

We need TurboCap• Typical Intel’s GigaNIC (e1000),
typical Dell PowerEdge2850 / Xeon 2.8GHz
RAM 1GB (PC3200, DDR2, 400MHz)• Threadshould is 140Mbps in Frame size = 64
• Frame size = 200 , actual rate 400Mbps• Frame size = 1500 , may be ok, no problem.
• We need TurboCap

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp

Debugging Environment• Using TurboCap, MMMM packets received by theapplication NNNN packets accepted by the filter anddumped to disk
• To fix, Optimize I/O access flow
packet -> IRQ -> SVC -> driver -> OS• Use 6 cores Xeon-L5640 and 24GB RAM !
( power resolve things and no page files )• Stop tcpdump and create program using pcap
libraries in C/C++ ( dumpcap.exe )• Pcap -> standard output -> FIFO -> SQLite• 3 month no problem
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp

Driving 250GB pcap file with Pilot• We use 250GB pcap file,huge huge file with Cascade
PilotPE installed into NotePC• Use view to check macro analysis,
and finally check the actual pcaps
using Wireshark• Only, best, easiest way todrive huge pcap file
Ikeriri network service co., ltd
http://www.ikeriri.ne.jp

1年保守なしQA and Demonstration

Ikeriri network service co., ltd
http://www.ikeriri.ne.jp48

Thank You
ikeriri network service

http://www.ikeriri.ne.jp(C) Ikeriri network service
http://www.ikeriri.ne.jp!!