hemu1438/forensicsgpt-training-data huggingface dataset:
The sleuth kit -
The Sleuth Kit (TSK) is a collection of UNIX-based command line tools that allow you to investigate a computer. The current focus of the tools is the file and volume systems and TSK supports multiple file systems.
Autopsy is a front-end for TSK which allows browser-based access to the TSK tools.
Features
The Sleuth Kit is arranged in layers. There is a data layer which is concerned with how information is stored on a disk and a metadata layer which is considered with information such as inodes and directories. The commands that deal with the data layer are prefixed with the letter d, which the commands that deal with the metadata layer are prefixed with the letter i.
Some of the commands in Sleuth Kit are:
blkcat Views the contents of a block.
blkls Lists unallocated blocks. Makes keyword searches more efficient. Gets a list of unallocated blocks.
blkcalc Tells you where an unallocated blocks are.
blkstat Details about a given block.
icat View contents of a file given its inode value or cluster number. Doesn't list directories, lists the contents.
ils Lists the files extents on a disk.
istat Information about an inode number.
Supported File Systems
NTFS
FAT-12, FAT-16, FAT-32, exFAT
ext2, ext3, ext4
UFS
ISO 9660 version 1
HFS+, HFSX
Yaffs
Note that several several of the file systems supported have known shortcomings. Check issue tracker for details.
File Search Facilities
Lists allocated and unallocated files.
Lists and sorts by file type.
Shows a time of creation and change.
Historical Reconstruction
fls and ils can be used to create a full listing of file system timestamps. The output of these commands can be inputted into mactimes which will generate a timeline of the file system timestamps.
Note that there are numerous known issues with the Body file format used by fls and ils.
Searching Abilities
Searches for keywords.
Builds an index.
Hash Databases
Uses MD5 or SHA-1.
Interfaces with NIST NSRL, Hashkeeper and customer databases.
Evidence Collection Features
Tracks forensic activity.
History
License Notes
The Sleuth Kit uses a mix of various licenses. Its core library, libtsk, is predominantly licensed under IBM Public License version 1.0 and Common Public License version 1.0. For more information see: https://github.com/sleuthkit/sleuthkit/blob/develop/licenses/README.md
Ext4 support
In 2011 Willi Ballenthin provided patches for the SleutKit to add ext4 support. These patches were integrated by Kevin Fairbanks into a separate fork of the Sleuth Kit. This fork was integrated in the 4.1.0 version.
Note that ext4 format features introduced after SleutKit 4.1.0 might not be supported and SleutKit tools might incorrectly represent these 1.
See Also
The Sleuth Kit How-To
Body file
tsk-cp
The mmls OCFA treegraph API example module.
External Links
Autopsy website
Fileobject -
fileobject is a Digital Forensics XML tag which is used to describe information about a file.
The file object can contain information about:
The file's name
The file's hash code(s)
The file's location on the disk.
Embedded metadata
Block hashes, a Bloom Filter, or a Similarity Digest for the file.
Other objects can be embedded in a fileobject object:
The byte_runs object specifies where the file is located on the disk.
A sector_hash object is a list of sector hash codes.
The sector_hash object could contain a nsrl_bloom object, which would be a bloom filter that contains all of the sector hashes.
XML Sample
<fileobject> <filename>README.txt</filename> <id>2</id> <filesize>43</filesize> <partition>1</partition> <alloc>1</alloc> <used>1</used> <inode>6</inode> <type>1</type> <mode>511</mode> <nlink>1</nlink> <uid>0</uid> <gid>0</gid> <mtime>1258916904</mtime> <atime>1258876800</atime> <crtime>1258916900</crtime> <byte_runs> <run file_offset='0' fs_offset='37376' img_offset='37888' len='43'/> </byte_runs> <hashdigest type='md5'>2bbe5c3b554b14ff710a0a2e77ce8c4d</hashdigest> <hashdigest type='sha1'>b3ccdbe2db1c568e817c25bf516e3bf976a1dea6</hashdigest> </fileobject> 
XML Tag	Meaning		
	Every file is inside a 		
YES	YES means that the file is an ""orphan,"" with no file name.		
3210	The file size in bytes.		
1	A "1" means that the file was not allocated in the file system. This may mean that the file was deleted.		
1	Not sure what this means.		
1114172320	The file's modification time, as a Unix timestamp (number of seconds since January 1, 1970 UTC).		
1195819392	The file's inode's creation time, as a Unix timestamp.		
1195794000	The file's access time, as a unix timestamp.		
121130496:3210	The file's fragments. Each fragment is represented as the byte offset from the beginning of the disk image (the first byte is byte #0) and a number of bytes.		
1	The number of fragments in the file.		
c27c0730b858bc60c8894300a98bba55	The file's MD5, as a hexadecimal hash.		
0277680d624e609f23aec9e4265c2d7d24bd3824	The file's SHA1, as a hexadecimal hash.		
1	The partition number in which the file was found.
Libdnet -
Abstract
libdnet provides a simplified, portable interface to several low-level networking routines, including
network address manipulation
kernel arp(4) cache and route(4) table lookup and manipulation
network firewalling (IP filter, ipfw, ipchains, pf, PktFilter, ...)
network interface lookup and manipulation
IP tunnelling (BSD/Linux tun, Universal TUN/TAP device)
raw IP packet and Ethernet frame transmission
Supported languages
C, C++
Python
Perl, Ruby (see below)
Supported platforms
BSD (OpenBSD, FreeBSD, NetBSD, BSD/OS)
Linux (Redhat, Debian, Slackware, etc.)
MacOS X
Windows (NT/2000/XP)
Solaris
IRIX
HP-UX
Tru64
External Links
Net::Libdnet - Perl interface to libdnet
libpcap - portable packet capture library
winpcap - libpcap for Windows
pypcap - libpcap Python module
dpkt - fast, simple packet creation and parsing in Python
PktFilter - win32 service to configure the IPv4 filtering driver in Windows 2000/XP/Server 2003
Universal TUN/TAP driver - virtual point-to-point network tunnel device
Tunnel driver for Solaris 8 (sparc64)
References
All information on this page can be found at libdnet.sourceforge.net and is credited to Dug Song - dugsong+libdnet@monkey.org
Barnyard2 -
About
Barnyard2 is an open source interpreter for Snort unified2 binary output files. Its primary use is allowing Snort to write to disk in an efficient manner and leaving the task of parsing binary data into various formats to a separate process that will not cause Snort to miss network traffic.
Barnyard2 has 3 modes of operation:
1. batch (or one-shot), 2. continual, and 3. continual w/ bookmark.
In batch (or one-shot) mode, barnyard2 will process the explicitly specified file(s) and exit.
In continual mode, barnyard2 will start with a location to look and a specified file pattern and continue to process new data (and new spool files) as they appear.
Continual mode w/ bookmarking will also use a checkpoint file (or waldo file in the snort world) to track where it is. In the event the barnyard2 process ends while a waldo file is in use, barnyard2 will resume processing at the last entry as listed in the waldo file.
The "-f", "-w", and "-o" options are used to determine which mode barnyard2 will run in. It is legal for both the "-f" and "-w" options to be used on the command line at the same time, however any data that exists in the waldo file will override the command line data from the "-f" and "-d" options. See the command directives section below for more detail.
Barnyard2 processing is controlled by two main types of directives: input processors and output plugins. The input processors read information in from a specific format ( currently the spo_unified2 output module of Snort ) and output them in one of several ways.
History
Barnyard is a critical tool for the parsing of Snort's unified binary files, processing and on-forwarding to a variety of output plugins. Unfortunately it has not seen an updated in over 4 years and is not going to be maintained by the original developers. With the new version of the unified format (ie. unified2) arriving we need something to bridge this gap. To quote directly from the Snort FAQ:
"Barnyard is an output system for Snort. Snort creates a special binary output format called unified. Barnyard reads this file, and then resends the data to a database backend. Unlike the database output plug-in, Barnyard is aware of a failure to send the alert to the database, and it stops sending alerts. It is also aware when the database can accept connections again and will start sending the alerts again."
The SXL team love barnyard. So much so that we want it to stay and have been tinkering around with the code to give it a breath of new life. Here is what we have achieved to far for this reinvigorated code base: Parsing of the new unified2 log files.
Maintaining majority of the command syntax of barnyard.
Addressed all associated bug reports and feature requests arising since barnyard-0.2.0.
Completely rewritten code based on the GPLv2 Snort making it entirely GPLv2.
Sources
Github About page
securixlive.com
Sniffer -
Sniffer (network sniffer) is a tool that can intercept, log and sometimes parse traffic passing over a network or part of a network.
Common Uses
Analyze network problems;
Detect intrusion attempts;
Monitor network usage;
Spy on other users and collect sensitive information such as passwords.
Detection of network sniffers
Passive sniffing
Generally, it is impossible to detect passive sniffing programs.
Detecting promiscuous mode
It is possible to detect network interfaces in promiscuous mode by sending requests (ICMP, ARP, etc) with destination IP address of a suspect machine and wrong destination MAC address. Network interfaces in promiscuous mode will pass this request and a suspect machine will reply (network interfaces in non-promiscuous mode will drop this packet).
Detecting reverse DNS lookup requests
Some sniffing programs do automatic reverse DNS lookups on the IP addresses they see. It is possible to detect such programs by correlating network traffic and reverse DNS lookup requests.
Detecting network sniffers using a honeypot (monitoring connections to fake accounts)
Active sniffing
Many sniffing programs provide special techniques to intercept traffic on a switched network:
ARP spoofing;
Fake DHCP server;
ICMP redirection.
As well as the ability to sniff encrypted data:
Man-in-the-middle attacks.
These active sniffing techniques can be detected in various ways.
Cheating network sniffers
IP fragmentation
Some sniffing programs cannot handle IP fragmentation correctly.
Shortcomings in TCP reassemblers
It is possible to cheat some TCP reassemblers by sending TCP packets with low IP TTL values (this TCP packet may not reach the destination host, but will be analysed by a network sniffer). This will break the resulting TCP stream.
Encryption: VPN tunnels, SSH tunnels, Tor.
Hidden channels
Sniffers
tcpdump
Wireshark
Xplico
NetworkMiner
Cain & Abel
ettercap (unsupported, last version - 2005/05/29)
dsniff (obsolete, last stable version - 2000/12/17)
justniffer
Bsd -
Berkeley Software Distribution (BSD) is one of the three major derivates of the Unix operating system from which many versions have been created such as OpenBSD, FreeBSD, NetBSD, Trusted BSD, and very early versions of Sun Microsystems’ Solaris. BSD was first released in the late 1970s and continues to have a strong following today exhibited in the continued existence of the above versions. The only exception is Sun who switched the design of their operating system over to System V Release 4.
Os fingerprinting -
OS fingerprinting is the process of determining the operating system used by a host on a network.
Active fingerprinting
Active fingerprinting is the process of transmitting packets to a remote host and analysing corresponding replies.
Passive fingerprinting
Passive fingerprinting is the process of analysing packets from a host on a network. In this case, fingerprinter acts as a sniffer and doesn't put any traffic on a network.
Fingerprinting techniques
Almost all fingerprinting techniques are based on detecting difference in packets generated by different operating systems.
Common techniques are based on analysing:
IP TTL values;
IP ID values;
TCP Window size;
TCP Options (generally, in TCP SYN and SYN+ACK packets);
DHCP requests;
ICMP requests;
HTTP packets (generally, User-Agent field).
Other techniques are based on analysing:
Running services;
Open port patterns.
Limitations
Many passive fingerprinters are getting confused when analysing packets from a NAT device.
Tools
Active fingerprinters:
Nmap
Passive fingerprinters:
NetworkMiner
p0f
See Also
NAT detection
Links
Remote OS detection paper
Tcp timestamps -
TCP timestamps are used to provide protection against wrapped sequence numbers. It is possible to calculate system uptime (and boot time) by analyzing TCP timestamps (see below).
These calculated uptimes (and boot times) can help in detecting hidden network-enabled operating systems (see TrueCrypt, linking spoofed IP and MAC addresses together, linking IP addresses with Ad-Hoc wireless APs, etc.
Supported Operating Systems
BSD/OS
FreeBSD, but not the default configuration in versions 3 to 4.3
HP-UX, recent versions
IRIX
Linux, kernel 2.1 and later
NetApp NetCache
Solaris 2.6 and later
Windows 2000, 2003, XP and Vista
Limitations
Some operating systems do not send TCP timestamps unless incoming TCP SYN packets will have this option enabled.
Method
Find all TCP packets with timestamp option (in Wireshark use following display filter: tcp.options.time_stamp);
Calculate target's clock frequency (e.g. 100 Hz or 1000 Hz) by analyzing two (or more) TCP timestamps in a certain period of time;
Use this frequency to calculate uptime.
Following tools can automate this process:
Nmap
Links
RFC 1323
https://uptime.netcraft.com/
Bzip2 -
The bzip2 (.bz2) file consists of a single bzip2 stream. The bzip2 stream consists of:
The stream header.
The stream header is 4 bytes in size and contains:
Offset
Size
Value
Description
0
	
2
	
"BZ"
	
Signature (magic number)
	
2
	
1
		
Version
'h' for Bzip2 ('H'uffman coding), '0' for Bzip1 (deprecated)
	
3
	
1
		
Block size
Value is defined in increments of 100 kB
'1'..'9' block-size 100 kB-900 kB (uncompressed)
Note: currently assumed that kB should be kiB
	
followed by zero or more compressed blocks
.compressed_magic:48 = 0x314159265359 (BCD (pi)) .crc:32 = checksum for this block .randomised:1 = 0=>normal, 1=>randomised (deprecated) .origPtr:24 = starting pointer into BWT for after untransform .huffman_used_map:16 = bitmap, of ranges of 16 bytes, present/not present .huffman_used_bitmaps:0..256 = bitmap, of symbols used, present/not present (multiples of 16) .huffman_groups:3 = 2..6 number of different Huffman tables in use .selectors_used:15 = number of times that the Huffman tables are swapped (each 50 bytes) *.selector_list:1..6 = zero-terminated bit runs (0..62) of MTF'ed Huffman table (*selectors_used) .start_huffman_length:5 = 0..20 starting bit length for Huffman deltas *.delta_bit_length:1..40 = 0=>next symbol; 1=>alter length { 1=>decrement length; 0=>increment length } (*(symbols+2)*groups) .contents:2..∞ = Huffman encoded data stream until end of block 
immediately followed by an end-of-stream marker containing a 32-bit CRC for the uncompressed data.
.eos_magic:48 = 0x177245385090 (BCD sqrt(pi)) .crc:32 = checksum for whole stream .padding:0..7 = align to whole byte 
The compressed blocks are bit-aligned and no padding occurs.
See also
cpio
gzip
tar
xz
External Links
Wikipedia: bzip2
Sqlite database format -
SQLite databases are used by many programs including several forensics tools, e.g. Autopsy 3. SQLite 3 is current and older SQLite packages cannot use sqlite3 databases so use sqlite3 tools.
SQLite3
SQLite version 3 uses a page-based storage where the pages are used for various types of data e.g. there are:
lock-byte pages
freelist pages
freelist trunk pages
freelist leaf pages
B-tree pages
table B-tree interior pages
table B-tree leaf pages
index B-tree interior pages
index B-tree leaf pages
payload overflow pages
pointer map pages
Write-Ahead Log (WAL)
The default method by which SQLite implements atomic commit and rollback is a rollback journal. In version 3.7.0 a "Write-Ahead Log" option was added.
Temporary sqlite files
Seen in e.g.
/Users/%USERNAME%/AppData/Local/Temp/etilqs_%RANDOM% 
Where "etilqs" is "sqlite" in reverse
Use Cases
Web Browser Data
Mozilla Firefox and Google Chrome both use SQLite version 3 databases for user data such as history, downloaded files.
Mobile OS
Google Android and Apple iOS use SQLite3 databases for many system applications. Phone data including calls, messages, and credentials are all stored in SQLite3.
External Links
Recovering Deleted Records from an SQLite database
Forensic examination of WAL files
Forensic Analysis of SQLite Databases: Free Lists, Write Ahead Log, Unallocated Space and Carving, by Belkasoft
The SQLite Database File Format, by the SQLite project
Write-Ahead Logging, by the SQLite project
Carving SQLite databases from unallocated clusters, by Richard Drinkwater, April 27, 2011
Recovering Data from Deleted SQLite Records: Redux, by John Lehr, September 13, 2013
Tools
Forensic Toolkit for SQLite
SQLite Forensics Explorer
Belkasoft Evidence Center
SQLite
SQLite Recovery
SQLite Forensic Reporter
SQLite Forensics
Olm viewer -
OLM Viewer is a freeware program built to offer a preview of Mac Outlook OLM files on a Windows Machine
Features
Emails, calendars, along with other items can be viewed from Outlook 2011 OLM file on Windows platform using this utility. Further, no particular file limitation is applicable on the size of OLM file. In case of a large number of emails, a search option can be applied to look up for a particular email.
Note: All Windows Versions up to 8.1 are supported for running the software appropriately.
External Links
Official Website
Thumbnails -
Thumbnails are reduced-size versions of pictures, serving the same role for images as a normal text index does for words.
Windows
See Thumbs.db.
Windows Vista
See Windows thumbcache
Thumbs.db no longer exists in Vista. This data has been moved to \Users\\AppData\Local\Microsoft\Windows\Explorer
Windows Vista will save thumbnails for files on mounted encrypted file systems (except EFS.
KDE & GNOME
KDE and GNOME are popular desktop environments for Linux and UNIX platforms. They are storing thumbnails in \~/.thumbnails.
Example thumbnail in GNOME:
$ hachoir-metadata .thumbnails/normal/0d97afdc637ac86d75d13e72172dc77c.png Metadata: - Image width: 128 pixels - Image height: 122 pixels - Bits/pixel: 24 - Pixel format: RGB - Compression rate: 1.6x - Compression: deflate - Producer: GNOME::ThumbnailFactory - Comment: Thumb::Image::Width=779 - Comment: Thumb::Image::Height=744 - Comment: Thumb::URI=file:///media/truecrypt1/123.jpg - Comment: Thumb::MTime=1216153400 - MIME type: image/png - Endian: Big endian 
GNOME will save thumbnails for files on mounted encrypted filesystems.
Example thumbnail in KDE:
$ hachoir-metadata .thumbnails/normal/2e4bc7bf05bd57e19c6f2a01cf4a8f71.png Metadata: - Image width: 96 pixels - Image height: 128 pixels - Bits/pixel: 24 - Pixel format: RGB - Compression rate: 1.6x - Compression: deflate - Producer: KDE Thumbnail Generator - Comment: Thumb::MTime=1236952299 - Comment: Thumb::Mimetype=image/jpeg - Comment: Thumb::Size=827837 - Comment: Thumb::URI=file:///home/fuf/10032009(002).jpg - MIME type: image/png - Endian: Big endian 
External Links
Ole compound file -
The Object Linking and Embedding (OLE) Compound File (CF) is used in other file formats as its underlying container file. It allows data to be stored in multiple streams.
The OLECF is also known as:
Compound Binary File (current name used by Microsoft
Compound Document File (name used by OpenOffice)
OLE2 file
MIME types
Because the OLECF by itself is just a container it does not use a mime type. A mime type assigned to an OLECF refers to its contents.
File signature
The OLECF has the following file signature (as a hexadecimal byte sequence):
d0 cf 11 e0 a1 b1 1a e1 
For earlier beta version of the format the following signature was used:
0e 11 fc 0d d0 cf 11 0e 
The OLECF has no distinct footer.
Contents
The OLECF uses a FAT-like file system to define blocks that are assigned to the stream using multiple allocation tables. It uses a directory structure to define the name of the streams.
The OLECF is used to store:
Microsoft Office 97-2003 documents:
Word Document (doc)
Excel Spreadsheet (xls)
Powerpoint Presentation (ppt)
MSN (Toolbar) (C:\Documents and Settings\%USERNAME%\Local Settings\Application Data\Microsoft\MSNe\msninfo.dat)
Jump Lists
StickyNotes.snt
Thumbs.db
Windows Installer (.msi) and patch file (.msp)
Windows Search (srchadm.msc)
External Links
Compound Binary File Specification, by Microsoft. Be warned this file contains at least one error: the directory entry name length is a size in bytes not in characters.
MS-CFB: Compound File Binary File Format, by Microsoft
Microsoft Compound Document File Format, by OpenOffice.org
OLE Compound File format specification, by the libolecf project
Office VBA
MS-OVBA: Office VBA File Format Structure, by Microsoft
Tools
libolecf
MiTec Structured Storage Viewer
oletools
Microsoft office -
EventLogs
As of Office 2010 related Office Alerts EventLog:
C:\Windows\System32\winevt\Logs\OAlerts.evtx 
See Also
Microsoft Office File formats
Microsoft Outlook
External Links
MS Excel 2013 Last Saved Location Metadata, by Jason Hale, January 12, 2014
Understanding the Microsoft Office 2013 Protected View Sandbox, by Yong Chuan, Koh, August 6, 2015