Package: pcap-parallel
pcap-parallel: split and parallel process a PCAP fileAboutThis package enables processing a PCAP file (for example, one produced
bytcpdump) to be processed in parallel on a multi-core machine.  It
achieves this by reading the entire file using thedpktmodule to
scan only each packet as quickly as possible to identify where the
packet boundaries are within the file.  Then sections of the file are
loaded intoio.ByteIOsections and handed to callback routines for
processing.  The callback routines are spun off as separate processes,
enabling them to do deeper (aka slower) packet processing.Code:https://github.com/hardaker/pcap-parallelNotesIt loads the entire file into memory!!  You've been warned.TODO: offer using file pointers instead with a length to readYour code must not care about packet ordering since parts of the
module will be processed by one function, and a future part by
another even though a TCP stream or similar might be split across
the multiple calls.It will attempt to calculate a split size and a maximum_cores value
for you, but it will not do a good job (especially on compressed
files).  You may (should) specify your own values when creating a
class.This will not a huge speed benefit if you aren't doing fairly
complex processing (the example below only does minimal processing).
If you're using something likescapy, though, it will definitely
help.It can handle compressed files (gzip, bz2, and xz) assuming you have
the needed decompression modules installed.It returns a list ofFutureobjects, so make sure to call.result()on each item in the list in order to ensure you get the
actual results from your callback.Because the results are run within a separate process, the contents
to return from each callback should be pickleable.Installationpip install pcap-parallelUsageThe following example uses thedpktmodule to count all the source
IP addresses seen in a PCAP file and display the results.  Note that
this is not super intensive processing, but at least demonstrates how
the module should work.importdpktimportipaddressfrompcap_parallelimportPCAPParallelfromcollectionsimportCounterdefprocess_partial_pcap(file_handle):"""Process a chunk of a larger PCAP fileNote: this will be launched multiple times in separate processes"""# store counters of datasrcs=Counter()# read the pcap in and count all the sourcespcap=dpkt.pcap.Reader(file_handle)fortimestamp,packetinpcap:eth=dpkt.ethernet.Ethernet(packet)ifisinstance(eth.data,dpkt.ip.IP):try:ip=eth.datasrcs[str(ipaddress.ip_address(ip.src))]+=1exceptException:passreturnsrcsps=PCAPParallel("test.pcap",callback=process_partial_pcap,)partial_results=ps.split()# merge the resultstotal_counts=partial_results.pop(0).result()forpartialinpartial_results:next_counts=partial.result()forkeyinnext_counts:total_counts[key]+=next_counts[key]# print the resultsforkeyintotal_counts:print(f"{key:<30}{str(total_counts[key]):>8}")LicenseSee the [./LICENSE] file for the details of the Apache 2.0 license.AuthorWes Hardaker 
USC/ISIhttps://www.isi.edu/~hardakerAcknowledgmentsThis module is a spin-off of a larger research project of Wes
Hardaker's at USC/ISI that is funded by Comcast.  We thank Comcast for
their support in making this module possible.