SharkFest'17 US
the 10 Top Wireshark
Tips & Tricks
Sample trace and supplemental files are located

http://www.ikeriri.ne.jp/download/sharkfest
Megumi Takeshita
Packet Otaku | ikeriri network service co.,ltd
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

Megumi Takeshita, ikeriri network service
a.k.a. packet otaku since first Sharkfest
•
•
•
•
•

Founder, ikeriri network service co.,ltd
Wrote 10+ books of Wireshark and capturing
and network analysis.
Reseller of Riverbed Technology ( former
CACE technologies ) and Metageek,
Dualcomm etc. in Japan
Attending all Sharkfest 9 times
Translator of QT Wireshark into Japanese

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

23 Top 10 Wireshark TIPS & TRICKS
1.
2.
3.
4.
5.

Collect only troubled IEEE802.11 packet
Use of Exclude Display filter
Stream ID and follow TCP in 2 way
Visualize using I/O Graph
Capture packet periodicallyand merge for stats
6. Sort and stack the pcap file automaticallyand divide by each filter value
7. Decrypting TLS/SSL data without key pair
8. HTTP2 dissection
9. QUIC dissection
10. Visualize using ErasticSearch ＋ Kibana
SharkFest'17 US • Carnegie Mellon University • June 19-22, 20173

#1. Collect only troubled IEEE802.11 packet
Wireless pcapng file contains tons of other packets
On capturing specified channnel,
you can collect only troubled IEEE802.11 packet
such as Deauthentication and Disassociation.

5423 packet injust a minuteSharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#1. Collect only troubled IEEE802.11 packet
When you collect Deauthentication and Disassociation,
wlan.fc.type_subtype == 12 or wlan.fc.type_subtype==14
No display filter but capture filter is good for wirelesstroubleshooting ( especially in Tokyo‘s crowded air )tshark -i 1 -f “subtype deauth or subtype disassoc"(http://www.tcpdump.org/manpages/pcap-filter.7.txt

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#1. Collect only troubled IEEE802.11 packet
You can collect Deauthentication/Disassociation,
regardless of knowing the WPA2 key( management frames are sent in clear text )Using capture filter is also good for reducing pcapng size.

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#2.Use the exclude Display filter
If you have no idea to find packet you needed,
Use the exclude Display filter is the easiest andbest way, and do not forget right click and choose
Apply filter > „... and not selected „SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#2.Use the exclude Display filter• The direction („ip.src“ or „ip.dst“) is determined by
the position of right click in packet list pane.
• Applying exclude display filter many times...
((!(ip.src == 216.58.197.234)) && !(ip.dst ==172.217.25.234)) && !(ip.src == 172.217.25.234)...
• You may find the packets you have not recognized.
• Excluding MAC Address is also useable

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#2.Use the exclude Display filter
Practice
1. Open wireless.pcapng file
2. Choose „Broadcast“ in Destination row in packet
list pane, right click, apply filter, „not selected“
3. Choose „02:cc:37:11:8a:04“ in Destination row,right click, apply filter, ... „and not selected“
4. set WPA-PWD (pass phrase : SSID )sharkfest:sharkfest-ikeriri in IEEE802.11 wireless
LAN preferences, Decryption Keys...
5. Then we got the packets we wanted.
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#3 Stream ID and follow TCP in 2 way• Wireshark set sequence number of transaction of TCP,
UDP, SSL and HTTP. For example Wireshark find first
SYN packet, and set stream number.
• We can check flow using straem id using display filter.
tcp.stream udp.stream ssl.stream http.stream• It is very useful to understand follow TCP stream

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#3 Stream ID and follow TCP in 2 way• It is also useful to add columns of TCP
Sequence no (tcp.seq)
TCP Segment Len (tcp.len)
Next sequence no (tcp.nxtseq)
Acknowledgement no (tcp.ack)

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#3 Stream ID and follow TCP in 2 way• Enter ip.src== client/server IP address in display filter.
• You can understand the mechanizm of TCP( seq no + segment length = next sequence number )SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#3 Stream ID and follow TCP in 2 way• Statistics > Flow Graph,choose Displayed
packets in Show,choose TCP flow inflow type• It is another way tounderstand the flow of
TCP from a peer.
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#3 Stream ID and follow TCP in 2 way
Practice• Open telnet.pcapng• Set as columns ofSequence no (tcp.seq)
TCP Segment Len (tcp.len)
Next sequence no (tcp.nxtseq)
Acknowledgement no (tcp.ack)• Set display filter as ip.src
from Client and Server• Choose Statistics > Flow Graph,choose Displayed packets in Show,choose TCP flow in flow type
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#4 Visualize using I/O Graph• Wireshark GUI has 3 types of Graph feature.
Rich interface and customizable interface
I/O Graph ( set Y axis as bits, packets, and Y fields )TCP specified analysis ( OS level )TCP Graph ( time/seq, RTT, throughput, window scaling )Simple arrow diagram
Flow Graph ( TCP flow, general flow )I/O Graph

TCP Graph

Flow Graph

Rich and customizable interface

TCP specified analysis (OS level )Simple arrow diagram

bits

Time/seq

packets

Y fields

RTT

Throughput

Window
TCP flow
scaling
SharkFest'17 US • Carnegie Mellon University • June
19-22, 2017

Generalflow

#4 Visualize using I/O Graph
I/O Graph ( set Y axis as bits, packets, and Y fields )Bits (line)
bits/s and bytes/s

packets (histogram)
Frequency

Y fields
Time, Size, …

• To see performance and throughput,set Y axis to bit/sec ( line )• To see errors and counting the number of packet, set Y
axis to packet/sec ( histogram )• To see time or size or other value,
set Y axis to Y fields
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#4 Visualize using I/O Graph
Practice4-1 To see performance and throughput
1. Add Item
2. Set Display filter as
ip.host contains FQDN
(sofmap.com)
3. Set Style as Line
4. set Y axis to bit/sec
5. Check Interval 1 sec
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#4 Visualize using I/O Graph
Practice4-2 To see errors and counting
webbrowse.pcang
1. Copy current item
2. Add the item’s
Display filter as “...
and tcp.analysis.
retransmission”
3. Set Style as Bar
4. set Y axis to
Packets/sec
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#4 Visualize using I/O Graph
Practice4-3 To see time or size or other value,

webbrowse.pcang
1. Delete 2nd item
2. Set Y Axis as“MAX(Y Field)”
3. Set Y field as“http.time”
4. Copy item
5. Set Display filter as “dns.qry.name contains FQDN”
6. Set Y field as “dns.time”
compareSharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#5 Capture packet periodically and merge for stats.
• dumpcap/tshark/wireshark has the same option tocapture packets for long term,• At first check the interface index with –D option
dumpcap / tshark / wireshark –b (capture multiple files )duration:NUM
filesize:NUM
files:NUM

switch to next file after switch to next file
NUM secs
after NUM KB

ringbuffer: replace
after NUM files• The pcap files are created as
name_seq(5digit)_YYYYMMDDHHMMSS.pcapng
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#5 Capture packet periodically and merge for stats.
• Using duration option is good for statistics.
• You can set duration as you want to set Units for stats.
60(1 minutes) / 3600(1 hour) / 86400(1 day)• -a ( autostop ) option is also available for statistics
dumpcap / tshark / wireshark –a (autostop option )duration:NUM
filesize:NUM
Stop after NUM secs Stop after NUM KB

files:NUM
Stop after NUM files• After capturing, you just check the number of files,
then merge SharkFest'17
using wildcard
option of mergecap.
US • Carnegie Mellon University • June 19-22, 2017

#5 Capture packet periodically and merge for stats.

• dumpcap –b duration:60 and –a files:60 option means we capture
60 files ( 1 hour ) that each duration is 60 seconds ( 1 minute )• You can merge packets for statistics, just counting files.
• For example, you can merge files just using wildcard
Includes Stat_00001_...～
Stat_00061_...(60 files)
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#5 Capture packet periodically and merge for stats.
Practice• Check interface index using –D option• Set 6 as duration with –b, set 10 as files with –a option• Capture packet to 10 test.pcapng files each duration is 6 second• Merge 10 files ( 60 seconds ) using wildcard• Create stats using tshark –qz option.
conv,ip conv,wlan dns,tree endpoints,ip endpoints,tcp
endpoints,wlan expert follow,http hosts http2,tree http_req,tree
http_srv,tree io,phs io,stat
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#6 Sort and stack the pcap file automaticallyand divide by each filter value• There are tons of bored and time-wasting pcapng task
when we inspect and dissect packets using Wireshark• Here is a very simple web access pcap file that contains 3 hosts.
• You need to create pcap files filtered with each host.
• How can you do this ?

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#6 Sort and stack the pcap file automatically
Practice• Windows10 64bit version supports
bash shell• You can add bash and use another convenient shell command
e.x. LOW ( Linux on windows ) or CYGWIN support too.
• You can extend wireshark CUI command with these shell script
and process packet task automatically.
(1) Collect source host list from the pcapng files
Hint: tshark –T fileds and –e option,(2) Filter packets by source hostand create each filtered pcapng file.
Hint: tshark –Y ip.src and shell script ( for )SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#6 Sort and stack the pcap file automatically• Open Settings > Update & Security> For Developers and activate
Practice
the Developer Mode
• Open Control Panel > Adding and Deleting Programs > Turn
Windows Features On or Off and check Windows Subsystem forLinux (Beta)• Open cmd.exe and type bash, or lxrun /install /ySharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#6 Sort and stack the pcap file automatically(1) Collect source host list from the pcapng files
Practice• Wireshark display filter string
ip.src_host means source IP host• You can pick up source host list using“Tshark –T fields –e Display Filter String”
tshark returns output
stream with duplication

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#6 Sort and stack the pcap file automatically(1) Collect source host list from the pcapng files• You can sort output streamPracticeusing sort command at first,
then delete duplication using uniq command

Sort with
duplication
at first
Delete duplication with uniq command
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#6 Sort and stack the pcap file automatically(1) Collect source host list from the pcapng files• Piping “| sort | uniq” give aPracticelist without duplication• You can create temporal working file (hostlist.txt) using redirect atfirst, then delete duplication using uniq command

Working file that contains ip.src host list
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#6 Sort and stack the pcap file automatically(2) Filter packets by source host and create file
• Filter packets by source host and create each filtered pcapng file.
• The command means open webaccess.pcapng and apply display
filter “ip.src_host==192.168.100.127” then create a file( 192.168.100.127.pcapng )the display filter was applied.
• Let’s create batch jobs for this task
• We can use “for” and use hostlist.txt
for /f "delims=" %a in (hostlist.txt) do (TASK)Parameter that contains a linedelimiter (¥n return code)that
wasUniversity
read from
SharkFest'17 US • Carnegie
Mellon• Junehostlist.txt
19-22, 2017

#6 Sort and stack the pcap file automatically(2) Filter packets by source host and create file• Exit bash shell and change directory to Desktop
test “for /f "delims=" %a in (hostlist.txt) do (TASK)”
Delimiter (¥n return code)
Display a line that wasread from hostlist.txt

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#6 Sort and stack the pcap file automatically

Practice• Collect source ip host list from webaccess.pcapng
tshark.exe -r webaccess.pcapng -T fields -e ip.src_host | sort |uniq > hostlist.txt• Filter packets by source host and create pcap file that contains
the each hostfor /f "delims=" %a in ( hostlist.txt) do ( tshark -r
webaccess.pcapng -Y ip.src_host==%a -w filtered_%a.pcapng )SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#6 Sort and stack the pcap file automatically

Practice

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#6 Sort and stack the pcap file automaticallyCompare CLI with GUI

GUI

CLI

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#7 Decrypting TLS/SSL data without key pair
• There are many encrypted communication using TLS/SSL in today’s Internet.
• For example, HTTP2 is based on TLS/SSL• If you have a server’s private key with certification ( PEM format ),you can decrypt TLS/SSL trace file, but it is mere.
• So it is essential to decrypt TLS/SSL trace by session• Client creates nonce (random) and encrypt nonce by server’s public key, it is
Encrypted
encrypted premaster secretnonce
encrypt
premaster• Master secret is created by premaster secret,secretServer’sthen session key is created by master secretcalculatepublic key• If you have unencrypted premaster secret,
Masteryou can retrieve session key of the SSL/TLS session
secret• You need to leak your unencrypted premaster secret
Sessionfrom Chrome. SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017key

#7 Decrypting TLS/SSL data without key pair
1. Set system environment variable
SSLKEYLOGFILE=Path of the premaster secret
SSLKEYLOGFILE = C:¥Users¥megumi¥Desktop¥ssl.key

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#7 Decrypting TLS/SSL data without key pair2.

Capture packets using Chrome ( https://www.ikeriri.ne.jp/ ) ssl2.pcapng
1.

Check SSLKEYLOGFILE was generated

ssl2.pcapng

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#7 Decrypting TLS/SSL data without key pair
3. Set (Pre)-Master-Secret log filename in SSL preference
Add path of ssl.key in (Pre)-Master-Secret log filename

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#7 Decrypting TLS/SSL data without key pair

Check #23,28 in trace file ssl2.pcapng

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#8 HTTP2 dissection

HTTP/2.0 has already here
•In Windows10 age, major Web browser and websites such as
google services including Google Map and Gmail, Facebook,
Twitter, Yahoo and web services are ready.
•HTTP/2.0 uses with TLS and all traffic is encrypted.
•Please run the Chrome, Microsoft Edge, Safari andcapture major webpage, HTTP2 has already been here.

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#8 HTTP2 dissection

HTTP/2.0 uses SSL/TLS
•
•
•
•
•

HTTP/2.0 works on SSL/TLS connection
Open Chrome URL “chrome://flags/”
Set Disable to QUIC protocol in list box
Now Chrome prefer to use HTTP2
Start capture and open www.twitter.com in
Chrome and save twitter.pcapng• Open the Chrome and type
chrome://net-internals/#http2you can see the HTTP/2 sessions• Wireshark has the dissector of HTTP2.0
https://www.wireshark.org/docs/dfref/h/http2SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#8 HTTP2 dissection
Decryption setting of SSL/TLS
Open twitter.pcapng and set (Pre)-Master-Secret log filename
Twitter_unencrypted_premaster_secret in SSL preference

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#8 HTTP2 dissection
Stream ID is important to dissect HTTP2
HTTP/2.0 manages communications using Stream mechanism
HTTP/2.0 uses 1 tcp connection and many Stream ( virtual connection
channel ) that has id and priority

1 tcp connection used by HTTP/2.0
HTTP/2.0 request

WebbrowserStream(id 1)

Binary frame
Binary frame

HTTP/2.0 response
HTTP/2.0 request
Stream(id 2)

serverBinary frame

Binary frame

HTTP/2.0 response

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#8 HTTP2 dissection
Add stream ID row and filter http2Select packet 1 and open HTTP2 header, and select Stream Identifier(http2.streamid) and right click and select “apply as column”
Set http2 in Display Filter

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#8 HTTP2 dissection
check HTTP2 request header• Set http2.header.value == "GET“ (it means HEADER(http2.type == 1))• Check 39 packet and open HTTP2 header
•
•
•
•
•
•
•
•
•
•

HTTP2 are binary frame
Check major header field
Method HTTP method
Authority Host of the server
Scheme HTTP / HTTPSPath Path of the object
Accept accept document typeUser-agent browser information
Referrer last visited uri
CookieSharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#8 HTTP2 dissection
check HTTP2 response header• http2.header.value == "200“ (it means HEADER(http2.type == 1))• Check 49 packet and open HTTP2 header
•
•
•
•
•
•

HTTP2 are binary frame
Check major header field:status Status code
Content-encoding compression
Server Web server
content-type document type

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#8 HTTP2 dissection
more information
For more detail mechanism and information about HTTP2
Please refer my last year’s presentation at Sharkfest 2016
“Wireshark 2.0 Tips for HTTP and HTTP2 Analysis”

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#9 QUIC dissection
What is QUIC• TCP 3 way handshake needs more connection time.
( off course there is a TCP Fast Open )• TCP have to send new segment after old segment isconfirmed as received successfully ( Head of Line Blocking)• TCP resend timer calculation and receive window
mechanism are not adequate in today’s internet• Anyway, TCP is slow protocol, so SSL/TLS is slow,
and HTTP is also slow, so Google create new protocol
instead of TCP/(SSL/TLS)/(HTTP/HTTP2) model• QUIC ( Quick UDP Internet Connection ) comes.
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#9 QUIC dissection
capturing QUIC• QUIC ( Quick UDP Internet Connection ) is Internet-Drafts
https://datatracker.ietf.org/wg/quic/documents/• open Wireshark and start capturing
open www.google.com and I’m feeling lucky, stop capture.
Then save filtered pcapng as imfeelinglucky.pcapng• At this time we just see some UDP stream

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#9 QUIC dissection
check QUIC connection• Open the Chrome and type
chrome://net-internals/#quic
you can see the QUIC sessions• Please check Connection UID

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#9 QUIC dissection
Decode as QUIC
• Click udp packet and right click “Decode As…” andchoose non-well-known port as QUIC ( this time port 50270)• Wireshark recognize the port is used for QUIC

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#9 QUIC dissection
QUIC architecture• HTTP2 (eth:ip:tcp:ssl:http2)• QUIC(eth:ip:udp:quic:http2(Encrypted))
QUIC overlaps SSL/TLS+TCP on UDP
with Encryption, Authentication, Session establishment,
Flow control, Error correction and Congestion Control

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#9 QUIC dissection
QUIC Public Packet Header

0
1
2
3
4
8
9
10
11
12
+--------+--------+--------+--------+--------+-----++--------+--------+--------+--------+| Public |Connection ID (64)
...
| ->
|
QUIC Version (32)| ->|Flags(8)|(optional)||(optional)|+--------+--------+--------+--------+--------+-----++--------+--------+--------+--------+
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
+--------+--------+--------+--------+--------+--------+--------+--------++--------+--------+--------+--------+--------+--------+--------+--------+|
Diversification Nonce| -> |
Diversification Nonce Continued| |(optional)||(optional)|+--------+--------+--------+--------+--------+--------+--------+--------++--------+--------+--------+--------+--------+--------+--------+--------+
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
+--------+--------+--------+--------+--------+--------+--------+--------++--------+--------+--------+--------+--------+--------+--------+--------+|
Diversification Nonce Continued| -> |
Diversification Nonce Continued| |(optional)||(optional)|+--------+--------+--------+--------+--------+--------+--------+--------++--------+--------+--------+--------+--------+--------+--------+--------+
45
46
47
48
49
50
+--------+--------+--------+--------+--------+--------+QUIC Wire Layout Specification|
Packet Number (8, 16, 32, or 48)|
docs.google.com/document/d/1WJvyZflAO2pq77yOLbp9NsGjC1CHetAXV8I0fQe-B_U|(variable length)|+--------+--------+--------+--------+--------+--------+

Connection ID: This is an unsigned 64 bit statistically random number selected by the clientthat is the identifier of the connection. Because QUIC connections are designed to remain
established even if the client roams, the IP 4-tuple (source IP, source port, destination IP,
destination port) may be insufficient to identify the connection. For each transmission direction,
when the 4-tuple is sufficient to identify the connection, the connection ID may be omitted.

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#9 QUIC dissection
Add Connection ID row and filter quic
Select packet 1 and open QUIC header, and select CID (quic.cid)
and right click and select “apply as column” Set quic in Display Filter

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#9 QUIC dissection
Add Connection ID row and filter quic
•

•
•
•
•

Now Wireshark can not decrypt QUIC negotiation
but we can check and find traffic of QUICCreate flow graph with displayed packet,
You can find there are no 3 way handshake,
There are few RTT in this connection.
QUIC use sometimes no RTT for resumption
and sometimes a few RTT for starting connection
QUIC works if client IP and port changed,
because QUIC has NAT-Traversal mechanism

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
Wireshark dissection is the source of big pcapng analysis• I showed this demo at last Sharkfest Europe 2016
• Wireshark is almighty decoder, packet dissection is not only for trace file analysis
within Wireshark itself, but also for the source of big data analysis• We can recode everything in network using Wireshark, and export dissection result
as JSON, JSON connect Wireshark with big data analysis tools such as Elastic
search and Kibana.

+Live packet
Decode / dissection
•
•

JSON

Big data analysis
Full-text search

Visualize
Real-time analysis

JSON (JavaScript Object Notation ) is one of open source data format like XML, JSON is commonlyused by many web application, data exchange. Tshark can also output packets as JSON
Tshark JSON (EK) output can be used for open source data analysis tool, Elastic Search and Kibana.

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
Set up Elasticsearch with Kibana environment
1.
2.
3.
4.
5.
6.
7.
8.

Download JDK, Curl and install, and set system environment variable
Download Elasticsearch and start server, check http://localhost:9200
Convert packet dissection data into Elasticsearch friendly JSON file
Entry packet dissection data (JSON) in Elasticsearch and check data
Modify the mapping, re-entry packet dissection data (JSON)
Download Kibana and start server, check http://localhost:5601
Access Kibana and set index
Search packet in full-text, visualize the packet and enjoy big data !

This time we are skipping STEP1-2, 4-7
and just enjoy Erastic Search and Kibana
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
1.

Download JDK, Curl and install, and set system environment variable
JDK(Java Developer Kit) 8 u101

http://www.oracle.com/technetwork/java/javase/downloads/index.html
Download jdk-8u101-windows-x64.exe
Execute and start setup program

Curl ( web access command )https://curl.haxx.se/download.html
Download curl-7.50.3.cab
Extract cab file and copy into “Program Files”
C:¥Program Files¥curl-7.50.3¥i386¥curl.exe

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
1.

Download JDK, Curl and install, and set system environment variable

Control Panel > System > System setting > detail settings > environmental variable
set JAVA_HOME=C:¥Program Files¥Java¥jdk1.8.0_101
set Path=(current path);C:¥Program Files¥Java¥jdk1.8.0_101; C:¥Program Files¥curl-7.50.3¥I386

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
Download Elasticsearch and start server, check http://localhost:9200

https://www.elastic.co/downloads

C:¥Users¥megumi¥Desktop¥elasticsearch-2.4.1¥bin¥elasticsearch.bat

1.Access https://www.elastic.co/downloads
2.Download elasticsearch-2.4.1.zip
3.Extract zip and open bin folder
4.Execute elasticsearch.bat
5.Check “started” in command prompt
6.Open http://localhost:9200

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
3.

Convert packet dissection data into Elasticsearch friendly JSON filetshark -T ek –r stream.pcapng > packet.json
1: Read stream.pcapng, convert EK (Elasticsearch friendly JSON file)
2: Redirect output stream to a file named packet.json

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
4.

Entry packet dissection data (JSON) in Elasticsearch and check data

curl –XPOST url @filename : use POST method to send data to server url
curl -XPOST http://localhost:9200/_bulk --data-binary @packet.json
C:¥Users¥megumi¥Desktop>curl -XPOST
http://localhost:9200/_bulk --data-binary @packet.json{"took":5748,"errors":false,"items":[{"create":{"_index":"packets-2016-1009","_type":"pcap_file","_id":"AVenqQppWAnL3ZFeCCi","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"status":201}},...
,{"create":{"_index":"packets-2016-1009","_type":"pcap_file","_id":"AVenqQpqWAnL3ZFeCC0","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"status":201}}]}

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
4.

Entry packet dissection data (JSON) in Elasticsearch and check data

Access http://localhost:9200/_search?pretty ( _search means all index (pretty
output ) and check data entry{"took" : 40,"timed_out" : false,"_shards" : {"total" : 5,
"successful" : 5,
"failed" : 0},"hits" : {"total" : 19,"max_score" : 1.0,
"hits" : [ {"_index" : "packets-2016-10-09","_type" : "pcap_file","_id" : "AVenqQpqWAnL3ZFeC-Ck","_score" : 1.0,"_source" : {"timestamp" : "1474777554499",

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
4.

Entry packet dissection data (JSON) in Elasticsearch and check data

If you failed to entry data , use curl “curl -XDELETE http://localhost:9200/*”

Check data mappings, open browser “http://localhost:9200/_mapping?pretty”{}"packets-2016-10-09" : {"mappings" : {"pcap_file" : {"properties" : {"layers" : {"properties" : {"data-text-lines" : {"properties" : {"data-text-lines_text" : {"type" : "string"}}},"dns" : {"properties" : {"dns_dns_count_add_rr" : {"type" : "string"
},

}
}
}
},
"timestamp" : {"type" : "string"
}

}
}
}
}

"timestamp" : "1474777554499",}

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
5.

Modify the mapping, re-entry packet dissection data (JSON)
http://localhost:9200/_mapping?pretty
Save mapping as “packet_mapping.json”
Delete header { "packets-2016-10-09" :Modify mapping as

Check character/return code UTF-8 / LF
Delete “curl -XDELETE http://localhost:9200/*”
Enter mapping as “curl -XPOST
http://localhost:9200/packets-2016-10-09 --databinary @packet_mapping.json”
Re-enter original data
curl -XPOST http://localhost:9200/_bulk --databinary @packet.json
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
6.

Download Kibana and start server, check http://localhost:5601

https://www.elastic.co/downloads

C:¥Users¥megumi¥Desktop¥kibana-4.6.1-windows-x86¥bin¥kibana.bat

1.Access https://www.elastic.co/downloads
2.Download kibana-4.6.1-windows-x86.zip
3.Extract zip and open bin folder
4.Execute kibana.bat
5.Check “Kibana index ready”in prompt
6.Open http://localhost:5601

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
7.

Access Kibana and set index

1.Access http://localhost:5601
2.Check “index contains time-based events”
3.Set Index name or pattern as“packets-2016-10-09” or “packets-*”
4.Set Time-field name as “timestamp”
5.Click “Create” button

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
8.

Search packet in full-text, visualize the packet and enjoy big data !
1.At First, use time picker to select the time of
packets ( just use “Last 1 year” is a good way )2.Check histgram and left pane
3.Select layers.ip_ip_dst in left pane,
click “add” and click “Visualize”,see and savethe graph as name “IP ”

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
8.

Search packet in full-text, visualize the packet and enjoy big data !

1. Select field ip.ip_ip_src, visualize and save the visualization as “IP Source”
2. Select “Visualize”, “Pie chart”, and “From a new search”,
check “Select buckets type”, click “Split slices”,
select “Terms” in Aggregation list box,
choose field “layers.frame.frame_frame_len”,
Apply changes, save the visualization
as “Frame length”
3. Using “layers.frame.frame_protocol” andcreate pie chart, save as “Protocols”
4.Click “Dashboard” and set layout ofthese 4 Visualization
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

#10 Visualize using ErasticSearch ＋ Kibana
8.

Search packet in full-text, visualize the packet and enjoy big data !

Just a few step,
We can create grate visualizationof packets, and enjoy big data !!

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

FinishUse Wireshark
SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017

Thank you very much for your listening
Use Wireshark !

Thank you !
どうもありがとうございます！

SharkFest'17 US • Carnegie Mellon University • June 19-22, 2017