PyFlag - An advanced network forensic framework

Network forensics is an investigation technique looking at the network traffic generated by a system. PyFlag is a general purpose, open source, forensic package which merges disk forensics, memory forensics and network forensics. This paper describes the PyFlag architecture and in particular how that is used in the network forensics context. The novel processing of HTML pages is described and the PyFlag page rendering is demonstrated. PyFlag’s novel processing of complex web applications such as Gmail and other web applications is described. Finally PyFlag’s report generation capabilities are demonstrated. a 2008 Digital Forensic Research Workshop. Published by Elsevier Ltd. All rights reserved. 1. What is network forensics? Network forensics refers to the forensic analysis of captured network traffic. In the simplest case the traffic represents network communication exchanged by a party during the course of some activity. Network forensics allows us to make forensic determinations based on the observed traffic, which may be relevant in the course of an investigation. This paper deals with the technical capabilities of an advanced network forensic system. We deliberately do not deal with the legal aspects of obtaining the network capture in the first place. We assume that a full network dump is available for analysis as obtained by the tcpdump program (Tcpdump, 2007). Network capture files are typically named PCAP files after the library used to read them. Considering the analysis of network dumps in the forensic context, as with other forensic disciplines, the network forensic analyst attempts to reconstruct or understand events from the information observed in the network capture. The aim of the analysis is commonly to establish high level facts such as attribution, intent, identity, timelines and other information which may be relevant to the case. From a forensic point of view we are often more interested in high level deductions, than in low level network protocol information. For example, in a typical fraud case we might be interested in a file of interest which was sent from a computer over the network. We may be less interested, in the first instance, in how the file was sent (e.g., using email, web pages or instant messenger), then in the contents of the file itself. Furthermore, we want to apply the same well tested tools and techniques from the mature disk forensic field on files discovered in the network capture. For example, compare hashes on transmitted images with well known image sets, or perform a keyword search on network traffic. This top down approach is typical in digital forensics as a way to control and regulate the vast quantities of data present. This paper introduces PyFlag as an innovative network forensic platform. Initially PyFlag is compared to current open source tools. The general PyFlag architecture is introduced, and the major architectural components are outlined. The paper then examines some important examples of the network forensic components in detail illustrating the trade offs made in their design. In particular we look at the components required to facilitate decoding of web mail – from the stream reassembler, the HTTP dissector and finally HTML parsing. The rendering of HTML pages and web mails is further explored. E-mail addresses: scudette@gmail.com, Michael.Cohen@afp.gov.au 1 There are numerous jurisdictional rules and regulation with regard to obtaining lawful intercepts of network communications, which fall outside the scope of this work. Readers are encouraged to obtain relevant legal advice. ava i lab le a t www.sc iencedi rec t .com journa l homepage : www.e lsev ie r . com/ loca te /d i in 1742-2876/$ – see front matter a 2008 Digital Forensic Research Workshop. Published by Elsevier Ltd. All rights reserved. doi:10.1016/j.diin.2008.05.016 d i g i t a l i n v e s t i g a t i o n 5 ( 2 0 0 8 ) S 1 1 2 – S 1 2 0