Package: pysharkfeat
PysharkfeatPysharkfeatis a TLS encrypted traffic feature extraction tool from pcaps written in Python by using Wireshark's command linetshark.Pysharkfeat is derived from an academic research onmalicious encrypted traffic analysis. Compare with other feature extraction tools such asFlowmeterandJoy, Pysharkfeat is easier to setup and use while providing rich features.FeaturesParse a single pcap or directory to generate meta and statistical featuresExport features in JSON filesSupport loggingTraffic features include:Meta: 5-tuple(src ip, src port, dest ip, dest port, timestamp), duration, stream indexStatistical:Bidirectional packet len and inter-arrival-time sum/max/min/mean/stdSPLT(Markov sequence of pkt len and time)Byte distribution, payload std and entropy.TLS: todo.Full features can be found infeat.pyor feature JSON file.EnvironmentLanguage: Python3.8, 3.9Dependence: WiresharkInstallationInstall pysharkfeat from pippip3 install pysharkfeatInstall Wireshark(tshark)Windows/Mac:https://www.wireshark.org/#downloadCentos:sudo yum install wiresharkUbuntu:sudo apt-get install wiresharkTest  tsharktshark --versionFor Windows, make sure tshark can be called by command line by addingtsharkto the environment path.Use casePysharkfeat can be used for machine learning research and threat analysis.There are several feature files intests/outputgenerated from pcaps atMalware Traffic Analysis, and you can immediately start analyzing them.ExampleThis code snippet can be found intests/demo.py.frompysharkfeat.featextractorimportFeatureExtractorimportjson,os# specify pcaps and output dirpcap_dir="./pcaps/2021-01-04-Emotet-infection-with-Trickbot-traffic.pcap"output_dir="./output"extractor=FeatureExtractor(pcap_path=pcap_dir,output_dir=output_dir)summary=extractor.main_extract_pcaps_feat()print(summary)# read feature filesfeat_file=os.path.join(output_dir,"2021-01-04-Emotet-infection-with-Trickbot-traffic.json")f=open(feat_file)stream_feats=json.load(f)forfeatinstream_feats:print("%s,  stream_index:%s,  byte dist entropy:%s"%(feat["pcap_name"],feat["stream_index"],feat["bd_entropy"]))# display stream index and byte distribution entropy features, and bd entropies are very close.2021-01-04-Emotet-infection-with-Trickbot-traffic.pcap,stream_index:3,bytedistentropy:7.9994647973149572021-01-04-Emotet-infection-with-Trickbot-traffic.pcap,stream_index:7,bytedistentropy:7.9031720995004422021-01-04-Emotet-infection-with-Trickbot-traffic.pcap,stream_index:9,bytedistentropy:7.9876935373284805...Performance considerationTimePysharkfeat is built on tshark, which may incur substantial overhead. The following table shows some test results on a Mac OSX(CPU i5, 16GB RAM).pcap namepcap sizenum of TLS streamstime(sec)2021-01-04-Emotet-infection-with-Trickbot-traffic.pcap5.4MB1010.82021-01-05-PurpleFox-EK-and-post-infection-traffic.pcap9.5MB811.52021-01-15-Emotet-epoch-1-infection-traffic.pcap5.9MB4038.22021-02-24-Qakbot-infection-with-spambot-traffic.pcap21.1MB94213.9StorageThe feature file of a single TLS stream has approximately 16KB. If a pcap has 100 TLS streams, the storage will be roughly 1.6MB.FeedbackYou are welcome to post a issue or feature request, or send email to the authorzliucd66@gmail.com.LicensePysharkfeat is open source and free to use under GPL V3 license. See LICENSE for more details.