SharkFest’19 US
Wireshark visualization
TIPS & tricks TOP10
Supplemental files
http://www.ikeriri.ne.jp/sharkfest/
and official site later
Megumi Takeshita
Packet Otaku, ikeriri network service#sf19us#sf19us •• UC
UC Berkeley
Berkeley •• June
June 8-13
8-13

Megumi Takeshita, ikeriri network service
•
•
•
•

•
•

Former CACE technologies reseller in 2008
Founder, ikeriri network service co., ltd
Wrote 10+ books about Wireshark
Reseller of Riverbed Technology and othercapture hardware/software in Japan
Attending all Sharkfest
One of contributor of Wireshark
Translate Wireshark into Japanese#sf19us • UC Berkeley • June 8-13

Visualization TIPS and TRICKS TOP10#1 Flow Graph#2 New Map#3 TCP Stream Graph#4 RTP Graph#5 IO Graph#6 Copy table values as CSV#7 Create statistics using tshark#8 Collect fields for Visualization#9 Export Packet dissection to JSON
#10 Splunk#sf19us • UC Berkeley • June 8-13

Part1
Wireshark

Part2tshark3

#1 Flow Graph with Conversation Filter• If you want to grab sequence, retransmission, andfragmentation between hosts, Flow Graph is a goodidea to visualize packets.
• Open trace file “sf19-1.pcapng” and choose Statistics> Flow Graph to create Flow Graph

Many hosts may be displayed#sf19us • UC Berkeley • June 8-13

4

#1 Flow Graph with Conversation Filter• Wireshark shows Flow Graph of all packets, there may
be tons of hosts in a Flow Graph, so use conversation
filter to focus between 2 hosts you want to.
• Choose a packet that you want to visualize
conversation, right click to choose “Conversation Filter”> “IPv4” to set display filter.
• Then click
Statistics > FlowGraph#sf19us • UC Berkeley • June 8-13

5

#1 Flow Graph with Conversation Filter
You can visualize
Between one conversation• Check “Limit to display filter” to limit conversation.
• You can visualize Flow Graph between 2 hosts.
#sf19us • UC Berkeley • June 8-13

6

#1 Flow Graph with Conversation Filter• If you want to see flow of TCP level connection,• Choose a packet and right click “Conversation Filter”> “TCP”, then select Statistics > Flow Graph, click“Limit to display filter” and change flow type as TCP.
• Time 2.351631 shows TCP retransmission and you canalso check the same Seq / Ack numbers.
Same Seq / Ack says
The sent segment is still
not ACKed and receive nosegment yet.
#sf19us • UC Berkeley • June 8-13

7

#2 New Map• Wireshark 3.x revived Map function and we canvisualize traffic by Map using Endpoints plugin.
• Open “sf19-2.pcapng” and click Statistics > Endpoints> UDP tab, then click Map > open in browser

IPv6 address rangeof Japan network
enabler (JPNE) for( tunneling)#sf19us • UCMAP-E
Berkeley• June 8-13

8

#2 New Map• Set “Cluster radius” slider to the right edge ( max ),then click blue dot to see UDP in entire Japan area.
• Set “Cluster radius” slider to the left edge (min),so you can find each address grouped by AS number.

AS
level

Countrylevel

This is a good way
to understand trafficby L4 protocols
geometrically, such ascountry and AS.

#sf19us • UC Berkeley • June 8-13

9

#3 TCP Stream Graph• Wireshark can list up all TCP/UDP connection using
Conversation table, so you can pick up slow connection,
create 5 types of TCP Stream Graph to visualize socket.
• Open “sf19-3.pcapng”, click Statistics >Conversation> TCP tab to list all TCP sockets and check Duration
column grey bar. (you can also sort the column)#sf19us • UC Berkeley • June 8-13

10

#3 TCP Stream Graph• Pick up the conversation which took 546.0800 duration.
• Sort again with Rel Start and count the stream ID(TCP stream starts with 0, and this connection is 4 )• Confirm the direction ( from B to A : downstream )We look for this slow
TCP connection( tcp.stream eq 4 )Press Graph after you
find stream index#sf19us • UC Berkeley • June 8-13

11

#3 TCP Stream Graph• Press Graph button to visualize TCP steam

Time / Sequence (Stevens) : understand stagnation
Time / Sequence (tcptrace) : understand stagnation as well as window size
Throughput : understand theoretical performance and segment length• You can drag/zoom, and refer each packet number
according to Wireshark main screen.
Time/Sequence(Stevens)

Time/Sequence(tcptrace)

Windowsize#sf19us • UC Berkeley • June 8-13

12

#3 TCP Stream Graph• Create Round Trip Time Graph and check RTT

Ave. RTT<1ms Fast Intranet ( may not think about TCP window mechanism
but you still need to think of Delayed ACK (40ms), Nagle and so on.
RTT<50ms Extranet or Domestic Internet (You may think of Retransmission
RTT>100ms International Internet or long range WAN links.
(You must think of TCP receive window control a.k.a LFN)

• Average round trip time is about
100ms in this time ( the plot of 0
ms just says there are no packet )so let’s create Window scaling
Graph to determine TCP RWIN

#sf19us • UC Berkeley • June 8-13

RTT=100ms is the threadshould13

#3 TCP Stream Graph• TCP window control mechanism works in big RTT
environment ( it takes long time to ACK, so we needbuffer for efficient conversation. ).
• There are enough margins of TCP window size
(Green RWIN vs Blue bytes out )• This trace file was capture in
old phone WAN link ( 128kbps )enough amount of
window that is not usedslow RTT and narrow bandwidth• You may think about TCP RWIN
in LFN ( Large Fat Network )#sf19us • UC Berkeley • June 8-13

14

#4 RTP Graph• Wireshark has Telephony menu to analyze VoIP,
SIP/RTP/RTCP packets and you can also listen too.
• Open “sf19-4.pcapng” and click Telephony >RTP Streams, and click a row of RTP stream and pressFind Reverse to select forward and reverse streams.
( or Shift + Click to select multiple rows )• Press Analyze button to see both direction at a glance#sf19us • UC Berkeley • June 8-13

15

#4 RTP Graph• Select Forward and Reverse tabs to investigate stream.
• Visualize RTP at a glance to press Graph tab.

Delta: <150ms OK <400ms Alert >400ms NG
Jitter: 20ns – 1 micro sec. ( as the case may be by Human)
Check Delta and Jitter#sf19us • UC Berkeley • June 8-13

16

#5 IO Graph• IO graph is common method to visualize traffic,
selecting adequate Y axis is very important.
1. Packet count graph : set Y axis by packets
2. Bandwidth graph : set Y axis by bits per seconds
3. Field value graph : choose math function to match.
4. Response time graph : set Y fields as http.time, etc.
• Open “sf19-5.pcapng” wireless trace file,and change profile to “customized IO Graph”• Click Statistics > IO Graph#sf19us • UC Berkeley • June 8-13

17

#5 IO Graph• What style is good for IO Graph ?
• If you want show the movement of the value,
set Line is good idea, and if you want to show theratio of partition, use Stacked Bar and Bar.
Line is good forunderstanding
the movement

Stacked bar at
the latter( backward)

Bar is easy tounderstanding
the value at the
momentStacked bar at
the first( front)#sf19us • UC Berkeley • June 8-13

18

#5 IO Graph

Set Y axis as packets to visualize counting frame by time.
For example Wireshark shows all packets vs TCP error
packets ( default ), Line is used by all packets, Bar isused for TCP error packets“set Y axis by packets” canvisualize counting frame by time
it is good for understanding
the ration of error, retransmission
and frame types.
#sf19us • UC Berkeley • June 8-13

19

#5 IO Graph
Left Graph shows the ratio of
wireless frame types, such as
management, control and data.
You can understand the statusof Wi-Fi
Stacked Bar forcounting

Right Graph shows the ratio ofdata frame and retransmitted
data frame. ( wlan.fc.retry==1)#sf19us • UC Berkeley • June 8-13

20

#5 IO Graph
2. Set Y axis by bits per seconds to visualize bandwidth
Bandwidth / throughput (bps)• set Y axis by bits
• Read Y axis as 10⁶ Mbps
• Compare CLIENT(ip.addr==192.168.100.135)
and Google traffic( ip.src_host contains google)• “set Y axis by bits” IO Graphis good for throughput#sf19us • UC Berkeley • June 8-13

21

#5 IO Graph3. set Y field and choose math function to visualize
specific field value• Add two items of TCP

RWIN and TCP segment.
• Set Y Field as theaverage of calculated
RWIN (tcp.window_size),and maximum ofsegment length (tcp.len)• Check TCP is OK or not.

#sf19us • UC Berkeley • June 8-13

22

#5 IO Graph
4. Response time graph : set Y fields as http.time, etc.
HTTP response time ( http.time )DNS response time ( dns.time )ICMP response time ( icmp.resptime )The RTT to ACK the segment( tcp.analysis.ack_rtt)• Add two items of HTTP
DNS response time.
• Set Y fields as Maximumof http.time and dns.time.
• Compare response time.
• Not HTTP but DNS is the
problem at the worst
case.

#sf19us • UC Berkeley • June 8-13

23

#6 Copy table values as CSV• IO Graph is very useful to visualize traffic, but
it uses only Time as X axis… We want to visualize
traffic using various dimension except for Time.
• Wireshark has various plugin table for statistics.
You can copy table values as CSV, then utilize themto Excel as Histograms and so on• Open “sf19-6.pcapng” that contains 10 mins wirelessclient traffic. And choose Statics > Endpoints,
and select IPv4 tab. Also check “Resolve Network
Address” from name resolution.
24#sf19us • UC Berkeley • June 8-13

#6 Copy table values as CSV• How about visualize host name as X axis ?
Check “Name resolution” and press Copy “as CSV”
and paste them to “sf19-6-1.txt”#sf19us • UC Berkeley • June 8-13

25

#6 Copy table values as CSV• Change extentions from txt to csv, start visualization
using Microsoft Excel or other apps.
• In this case, using Excel to create a new sheet.
• Open sample visualization example table file“sf19-6-1.xlsx”#sf19us • UC Berkeley • June 8-13

26

#6 Copy table values as CSV
Someone loves
www.jurassicworld.jp• Just cut Address,
Packets, and Bytes
Rows, then paste
another tab. Then
Insert > Graph tocreate Hisograms• Someone loves
Jurassic World
movie.

#sf19us • UC Berkeley • June 8-13

27

#6 Copy table values as CSV• Copy CSV to anothersheet, edit rows
following City, Packets
and Bytes. Then groupby City name, clicking
Data > subtotal• Set group by City,
count by Total of
Packets and Bytes
rows, then press OK#sf19us • UC Berkeley • June 8-13

28

#6 Copy table values as CSV• Press left sidegroup button [2].
• Copy City, Packets,
and Bytes row
subtotaled by Cityand paste values
into another sheet.
• Edit some cells tolimit top 100 data
#sf19us • UC Berkeley • June 8-13

29

#6 Copy table values as CSV• Create another tab and copy City, Bytes and Packets.
• Insert Graph> Bing Map• Press Filter
button to set
data region.
• You can see
packets anddata in Map#sf19us • UC Berkeley • June 8-1330

#7 Create statistics using tshark• Tshark is a CLI version of Wireshark, sotshark can use some statistic plugin with –qz option.
Check online help with “tshark –qz help”• The option of protocol hierarchy statistics chart is“io,phs” so open “sf19-7.pcapng” with “-qz io,phs”#sf19us • UC Berkeley • June 8-13

31

#7 Create statistics using tshark• We got protocol hierarchy statistics
of all protocols in text format.
• For making pie chart we need toprocess text data to match CSV.
• Remove “frames:” and “bytes:”
using sed -e 's/frames://' -e's/bytes://‘ using bash
• Redirect output stream as phs.csv
tshark -r sf19-7.pcapng -qz io,phs | sed -e's/frames://' -e 's/bytes://' >> phs.csv#sf19us • UC Berkeley • June 8-13

32

#7 Create statistics using tshark• tshark -r sf19-7.pcapng -qz io,phs | sed -e 's/frames://'-e 's/bytes://' >> phs.csv
you also may use “tr -d ' ‘” to remove space character• Open csv in Excel and create a new sheet and copy
from original data and remove unnecessary lines.
• Set Data>Delimiter as space and add header line
• Insert Graph > Donut Pie Chart and customize color,size, index, title, etc• Finally we can find UDP echo is the majority#sf19us • UC Berkeley • June 8-13

33

#7 Use tshark to create statistics
Nice donut,
Isn’t it ?

#sf19us • UC Berkeley • June 8-13

34

#8 Collect fields for Visualization• Tshark is a CLI version of Wireshark, as well as nicedata processing tool for visualization from trace files.
• Check –T option and you can pick up any fields ofdissector from trace file like –T fields –e ip.src• This time we want to collect host information of httpopen “sf19-8.pcapng” using tsharkand collect http.host field information as below
tshark –r sf19-8.pcapng –Y http.request –T fields–e http.host ( use –R read filter if huge trace file )#sf19us • UC Berkeley • June 8-13

35

#8 Collect fields for Visualization• The output contains host header information in eachhttp request, start data processing for visualization• At first we need bash and the typical technics below
sort an output stream, then count the same line,
and sort again for descending for top list
tshark -r sf19-8.pcapng -Y http.request -T fields -e
http.host | sort | uniq -c | sort –rn(sort alphabetically and count duplications )#sf19us • UC Berkeley • June 8-13

36

#8 Collect fields for Visualization
Sort descending ofCount / hostname• Redirecting the output stream as csv
tshark -r sf19-8.pcapng -Y http.request -T fields-e http.host | sort | uniq -c | sort -rn >> hostlist.csv• Open CSV file and set delimiter using Excel#sf19us • UC Berkeley • June 8-13

37

#8 Collect fields for Visualization• Set delimiter as space and create a new sheet,• copy and paste host and count rows into new sheet.
• Insert People graph and save as topwebsite.xlsx#sf19us • UC Berkeley • June 8-13

38

#8 Collect fields for Visualization• How about TLS ?
• Client Hello messages may
contain host nameas one of extensions(tls.handshake.type== 1)• Server name fields locates in
one of extentions in Client Hello(ssl.handshake.extensions_server_name )Server Name fields in TLS#sf19us • UC Berkeley • June 8-13

39

#8 Collect fields for Visualization• Set display filter as “tls.handshake.type == 1” andcollect fields of “ssl.handshake.extensions_server_name” intrace. So use the command in bash to create csv
tshark -r sf19-8.pcapng -Y ssl.handshake.type == 1
-T fields –e ssl.handshake.extensions_server_name| sort | uniq -c | sort -rn >> tlshostlist.csv• Note: sometimes only old filter string is accepted, so
we use ssl display filter word instead of tls in tshark#sf19us • UC Berkeley • June 8-1340

#8 Collect fields for Visualization• If you got blank in server
name, there is no hostinformation in Client Hello• Create People Graph inthe same way and savefile as toptlssite.xlsx#sf19us • UC Berkeley • June 8-13

41

#9 Export Packet dissection to JSON
• I talked about Visualization using Elastic Search andKibana from json file from Wireshark at Sharkfest’17+

Live packet
Decode / dissection

JSON

Big data analysis
Full-text search

Visualize
Real-time analysis• Wireshark 3.x / tshark now support many options to
output json file from trace file and live capture.
• -T json / jsonraw / ek (Elastic search Kibana )and we can also use –G elastic-mapping and--elastic-mapping-filter <protocols> option#sf19us • UC Berkeley • June 8-13

42

#9 Export Packet dissection to JSON+Big data analysis
Full-text search

Visualize
Real-time analysis

Setup Elastic andKibana environments

1.Check your machine supports JavaC:¥Users¥megumi>set | find "JAVA"
JAVA_HOME=C:¥Program Files¥Java¥jre1.8.0_212

2.Access https://www.elastic.co/jp/downloads
3.Download Elastic search, Kibana
4.Extract zip and open each bin folder
5.Execute elasticsearch.bat
6.Check “started” in command prompt
7.Open http://localhost:9200
8.Execute kibana.bat
9.Check “Kibana index ready”in prompt
10.Open http://localhost:5601#sf19us • UC Berkeley • June 8-13

43

#9 Export Packet dissection to JSON
http://localhost:9200
http://localhost:5601• This time I used old set of Elastic + Kibana
elasticsearch-2.4.1 and kibana-4.6.1-windows-x86#sf19us • UC Berkeley • June 8-13

44

#9 Export Packet dissection to JSON
Export packet dissection to JSON ( Elastic +Kibana ) format from sf19-9.pcapng
tshark –r sf19-9.pcapng –T ek > trace.json
Open editor and check json file#sf19us • UC Berkeley • June 8-13

45

#9 Export Packet dissection to JSON*OPTION*

If you we want to create json file including only tcp and
ip header, we can use –e tcp –e ip
tshark –r sf19-9.pcapng –T ek –e tcp –e ip
Check output to confirm the json file contains only tcp
and ip header information. Also –j/-J-j <protocolfilter>

protocols layers filter if -T ek|pdml|json selected(e.g. "ip ip.flags text", filter does not expand childnodes, unless child is specified also in the filter)-J <protocolfilter>
top level protocol filter if -T ek|pdml|json selected(e.g. "http tcp", filter which expands all child nodes)#sf19us • UC Berkeley • June 8-13

46

#9 Export Packet dissection to JSONPut trace.json into Elastic
curl -H "Content-Type: application/x-ndjson" XPOST http://localhost:9200/_bulk --data-binary@trace.json
Check “successful”#sf19us • UC Berkeley • June 8-13

47

#9 Export Packet dissection to JSON
• We success putting json file into Elastic, but data
schema ( term mapping in Elastic ) is not correct//• curl http://127.0.0.1:9200/_mapping
All fields types
are string
all fields types are recognized as “string”#sf19us • UC Berkeley • June 8-13

48

#9 Export Packet dissection to JSON
• When you create json file using tshark / Wireshark,
there are problems about mismatch of database
schema ( a.k.a. “mapping” in Elastic )When you upgrade Wireshark and some protocol
dissector is updated or modified,
the output json file format may be changed.
C:¥Users¥megumi¥Desktop>tshark --version
TShark (Wireshark) 2.4.2 (v2.4.2-0-gb6c63ae086)
tshark -T ek -r stream.pcapng >> json242.txt

C:¥Users¥megumi¥Desktop>tshark --version
TShark (Wireshark) 3.0.2 (v3.0.2-0-g621ed351d5c9)
tshark -T ek -r stream.pcapng >> json302.txt#sf19us • UC Berkeley • June 8-13

49

#9 Export Packet
dissection to JSON
• We can create adequate
Elastic mapping file semiautomatically using tshark• If we want to create flow
based schema information
including ip, tcp and udp
tshark -G elastic-mapping--elastic-mapping-filter
ip,tcp,udp > mapping.json#sf19us • UC Berkeley • June 8-13

Timestamp
Type: date
IP version
field
Type: short

Delay bit of
IP TOS field
Type: bool
Identification
field
Type: int
Destination
IP address
Type : ip50

#9 Export Packet dissection to JSON
• We need to delete all data and schema

curl -XDELETE http://localhost:9200/*• Then put mapping information into Elastic
curl -H "Content-Type: application/x-ndjson" -XPOST
http://localhost:9200/packets-2016-09-25 --data-binary@mapping.json• Check mapping “curl http://127.0.0.1:9200/_mapping”#sf19us • UC Berkeley • June 8-13

51

#9 Export Packet dissection to JSON
• Without mapping file

• With mapping file “mapping.json”

curl -H "Content-Type: application/x-ndjson" -XPOST
http://localhost:9200/packets-2016-09-25 --data-binary @mapping.json#sf19us • UC Berkeley • June 8-13

52

#9 Export Packet dissection to JSONPut trace.json again into Elastic
curl -H "Content-Type: application/x-ndjson" XPOST http://localhost:9200/_bulk --data-binary@trace.json
Check “successful”#sf19us • UC Berkeley • June 8-13

53

#9 Export Packet dissection to JSON
• Its time to use Kibana !
http://localhost:5601• Set index patternas packets-2016-09-25(may work packets-*)• Set Time-filed nameas timestamp ( type:date )• Click “Create”#sf19us • UC Berkeley • June 8-13

54

#9 Export Packet dissection to JSON
• Check mapping is correctly
assigned as adequate type.

layers.udp.udp_time_relative as datelayers.tcp.tcp_window_size_scalefactor as number
layers.tcp.tcp_options_scpsflags_bets as booletc.

• Enjoy Visualization#sf19us • UC Berkeley • June 8-13

55

#10 Splunk• Splunk is one of big data processing tools forvisualizing trace files via CSV or JSONhttps://splunkbase.splunk.com/app/2748/• We can use free if the data size is under 500MB
in Windows / Linux / macOS environments• There are two major way to convert pcap/pcapng
JSON

Live packet
Wireshark / tshark

you also import pcap itself

CSV

Pcap/pcapng#sf19us • UC Berkeley • June 8-13

56

#10 Splunk• There are sample trace files including huge packets.
(https://www.bettydubois.com/sharkfest19)• I use 1G trace (1G-1050000Pkts.pcapng ) that contains
about 1 million packets• Open the file in Wireshark( recommend with ReadFilter
or light profile for huge file )and Export Packet Dissections toexport CSV which contains justa packet summary information )#sf19us • UC Berkeley • June 8-13

57

#10 Splunk
In this case, we use the default information of
packet summary pane, such as Numbers, Source,
Destination, Protocol, Length and Info
Though you can off course customize them
Using tshark is also a good way to handle big trace files,
tshark -r 1G-1050000Pkts.pcapng -T text>> 1G-1050000Pkts.csv
or you can use –T json for your customized dissector fields
information ( with –e or –j or –J options )#sf19us • UC Berkeley • June 8-13

58

#10 Splunk• So input 1G-1050000Pkts.csv into Splunk,set fields name and indexes are created automatically• I’ll not talk about Splunk in detail , there are tons of
documents and samples you can refer• Open splunk page and login (http://localhost:8000/)
1. Click [Search and Reporting] in Left pane
2. Choose time range as all terms
3. Type “source="1g-1050000pkts.csv" | chart count bydestination and set style as pie chart#sf19us • UC Berkeley • June 8-13

59

#10 Splunksource="1g-1050000pkts.csv" | chart count by destination

source="1g-1050000pkts.csv" | chart count by protocol#sf19us • UC Berkeley • June 8-13

60

#10 Splunksource="1g-1050000pkts.csv" | top length

source="1g-1050000pkts.csv" | top Info#sf19us • UC Berkeley • June 8-13

61

Think Visually• You have finished visualization of trace file,then its turn to think visually.
• Stop looking each frame in detail,
Look over the traffic visually.
• You may find a new clue
which you have never found !!
• USE WIRESHARK
and THINK VISUALLY#sf19us • UC Berkeley • June 8-13

62

USE WIRESHARK

Thank you for attending !!
Please complete the SharkFest’19 US app-based survey

ikeriri network service

http://www.ikeriri.ne.jp#sf19us • UC Berkeley • June 8-13