Package: heifip
heiFIPstands for Heidelberg Flow Image Processor.
It is a tool designed to extract essential parts of packets and convert them into images for deep learning purposes.
heiFIP supports different formats and orientations.
Currently, we only supportofflinenetwork data analysis.
However, we plan to adapt our library to supportonlinenetwork data too to enable live-probing of models.Live NotebookLatest ReleaseSupported VersionsProject LicenseContinuous IntegrationTable of ContentsMain FeaturesMotivationExamplesGetting StartedBuilding from sourceCitationCreditsAuthorsLicenseMotivationThe idea to create heiFIP came from working with Deep Learning approaches to classify malware traffic on images. Many papers use image representation of network traffic, but reproducing their results was quite cumbersome. As a result, we found that there is currently no official library that supports reproducible images of network traffic. For this reason, we developed heiFIP to easily create images of network traffic and reproduce ML/DL results. Researchers can use this library as a baseline for their work to enable other researchers to easily recreate their findings.Main FeaturesDifferent Images: Currently, we support plain packet to byte representation, and flow to byte representation with one channel each. An image is created with same width and height for a quadratic representation.Flow Imagesconverts a set of packets into an image. It supports the following modifications:Max images dimensionallows you to specify the maximum image dimension. If the packet is larger than the specified size, it will cut the remaining pixel.Min image dimesionallows you to specify the minimum image dimension. If the packet is smaller than the specified size, it fills the remaining pixel with 0.Remove duplicatesallows you to automatically remove same traffic.Appendeach flow to each other or write each packet to a new row.Tiledeach flow is tiled into a square image representation.Min packets per flowallows you to specify the minimum number of packets per flow. If the total number of packets is too small, no image will be created.Max packets per flowallows you to specify the maximum number of packets per flow. If the total number of packets is too great, the remaining images are discarded.Packet Imageconverts a single packet into an image.Markov Transition Matrix Image: converts a packet or a flow into a Markov representation.Headerprocessing allows you to customize header fields of different protocols. It aims to remove biasing fields. For more details look intoheader.pyRemove Payloadoptions allows you to only work on header data.Fast and flexible: We rely onScapyfor our sniffing and header processing. Image preparation is based on raw bytes.Machine learning orientation: heiFIP aims to make Deep Learning approaches using network data as images reproducible and deployable. Using heiFIP as a common framework enables researches to test and verify their models.ExamplesImage TypeDescriptionExamplePacketConverts a single packet into a square image. Size depends on the total lengthFlowConverts a flow packet into a square imageMarkov Transition Matrix PacketConverts a packet into a Markov Transition Matrix. Size is fixed to 16x16.Markov Transition Matrix FlowConverts a flow into a Markov Transition Matrix. It squares the image based on the number of packetsGetting StartedInstall our package using PyPipipinstallheifipNow, you can use the integrate CLI:>fip
Usage:fip[OPTIONS]COMMAND[ARGS]...

Options:--versionShowtheversionandexit.-h,--helpShowthismessageandexit.

Commands:extractTo extract images from PCAPs, we currently split the command into flow and packet:>fipextract
StartingFlowImageProcessorCLI
Usage:fipextract[OPTIONS]COMMAND[ARGS]...

Options:-h,--helpShowthismessageandexit.

Commands:flowpacket# Show help information>fipextract[flow/packet]-h
StartingFlowImageProcessorCLI
Usage:fipextractflow[OPTIONS]Options:-w,--writePATHDestinationfilepath,storesresult[required]-r,--readPATH[required]-t,--threadsINTEGERNumberofparallelthreadsthatcanbeused[default:4]--preprocess[NONE|HEADER]Appliesapreprocessingtotheinputdata:none:Nopreprocessingpayload:Onlypayloaddataisusedheader:Preprocessesheaders(DNS,HTTP,IP,IPv6,TCP,UDPsupported)toremovesomebiasingdata[default:NONE]--min_im_dimINTEGERMinimumdimouputimagesneedtohave,0=Nominimumdim[default:0]--max_im_dimINTEGERMaximumdimouputimagescanhave,0=Nomaximumdim[default:0]--remove_duplicatesWithinasingleoutputfolderbelongingtoasingleinputfoldernoduplicateimageswillbeproducediftwoinputsleadtothesameimage--min_packetsINTEGERMinimumpacketsthataFlowImageneedstohave,0=Nominimumpacketsperflow[default:0]--max_packetsINTEGERMinimumpacketsthataFlowImageneedstohave,0=Nominimumpacketsperflow[default:0]--append--tiled--widthINTEGER[default:128]-h,--helpShowthismessageandexit.

>fipextractflow-r/PATH/PCAPs-w/PATH/IMAGESImport FIPExtractor to run it inside your program:extractor=FIPExtractor()img=extractor.create_image('./test/pcaps/dns/dns-binds.pcap')extractor.save_image(img,'./test/pcaps/dns/dns-binds.pcap')Building from sourceSimply run:pip install .Publications that use heiFIP[A Generalizable Approach for Network Flow Image Representation for Deep Learning] - CSNet 23[Explainable artificial intelligence for improving a session-based malware traffic classification with deep learning] - SSCI 23CreditsNFStreamfor the inspiration of theREADME.mdand workflow testing.AuthorsThe following people contributed to heiFIP:Stefan Machmeier: CreatorManuel Trageser: Header extraction and customization.LicenseThis project is licensed under the  EUPL-1.2License- see the License file for details