hemu1438/forensicsgpt-training-data huggingface dataset:
Live cd -
A live CD is a CD containing a bootable computer operating system. Live CDs are widely used in computer forensics and incident response.
Advantages
Physical memory of a computer can be imaged by performing cold boot attack without running tools on an untrusted operating system.
Acquisition over a network connection without running tools on an untrusted operating system.
No need to reconstruct RAID arrays;
etc.
Disadvantages
Out-of-date software;
No simple way to reconfigure Live CD: you cannot easily rebuild foo to support bar (e.g. rebuild The Sleuth Kit to support AFF.
See Also
Forensics Live CDs
Forensic Live CD issues
Word document (doc) -
The Word Document (DOC) file format has the .doc extension. This file type originates from Microsoft Word. However, other word processing software can be used to display these files as well. These include:
AbiWord
LibreOffice
OpenOffice
WordPad
WordPerfect
The Word DOC file format should not be confused with DOCX.
MIME types
The following MIME types apply to this file format:
application/msword
application/doc
appl/text
application/vnd.msword
application/vnd.ms-word
application/winword
application/word
application/x-msw6
application/x-msword
zz-application/zz-winassoc-doc
File signature
Microsoft Word documents of version 97-2003 use the OLE Compound File. These files therefore have the OLECF file signature
The object stream of the OLECF containing a Word document contains the string "Word.Document" with some version.
Word 97-2003 documents
The Word Binary File format is stored in the OLECF using multiple streams:
WordDocument stream
Table stream (0Table, 1Table)
Data stream
Encryption
Versions 97/2000 encrypt documents with a very weak algorithm. This password scheme can be broken easily by several different products and it is possible to decrypt the contents without discovering the password. This is done by testing all 1,099,511,627,776 possible keys. Ultimate Zip Cracker by VDGSoftware is one utility that can perform this decryption.
On a unix-like machine try this command to extract strings from a .doc file:
cat /tmp/test.doc | tr -d \\0 | strings | more
(where /tmp/test.doc is the path to your .doc file)
Note that a Word 97 and later document can contain both extended ASCII with codepage 1252 (codepage 1252 compressed text) and UTF-16 little-endian text. Word document can also contain 'East Asian' or 'Complex script' languages. Also the text stream contains information about all the parts of the Word document (header/footer, foot/endnote, annotation, etc.) Therefore using basic Unix string is very rough approach of finding data in a Word document. Use the wvtools or more sophisticated tools instead.
External Links
Word 97-2007 Binary File Format by Microsoft
Gethashes sh -
General Usage
One of the most important things to remember when performing digital forensics is how to preserve authenticity. Calculating and retaining hashes of the files you gather during your investigation (whether they be script output or actual binaries) is a great way to prove that your collected evidence hasn't been tampered with. Because of the risk of hash collisions, it is important to use multiple hashing algorithms for each file (e.g., MD5 and SHA1).
GetHashes.sh is a simple bash script that quickly calculates MD5 and SHA1 hashes for all files in the same directory as the script. This is much quicker and simpler than running the hashing commands manually for each file. The script is used by placing the script in the same folder as the files to hash, and then running the script. This will produce two folders of hashes, one of MD5, and one of SHA1 hashes. These can be used at a later time as evidence that the forensic data has not been tampered with. The script also ignores itself and any directories located in the same directory (to avoid errors when trying to hash directories). Only files in the same directory as the script are hashed.
Usage Instructions
In order to use the UNIX file hashing script, make sure you download the getHashes.sh file from the repository above onto a UNIX machine with md5sum and sha1sum installed (most UNIX-like operating systems have these installed by default). The guide below used a Kali Linux v1.0.9 machine.
Next, place the getHashes.sh script into the same directory as the files you want to hash. In the screenshot below, the script was placed in the same directory as a collection of text files containing the output of commands run on the victim machine, transferred to a forensic workstation using Netcat.
Next, open up terminal into the directory with the getHashes.sh script and execute the script.
NOTE: If you get permission errors when trying to execute the script, you likely need to run the following command in terminal to make the script executable:
chmod a+x getHashes.sh
As you can see in the screenshot above, the script created MD5 and SHA1 directories because they didn't yet exist in that directory. These new directories each contain .MD5 and .SHA1 files, respectively (each containing the hash and filename of its corresponding file), for all files in the same directory as the script. These hashes are also sent as output to the terminal--with the MD5 hash listed first and the SHA1 hash listed second--to simplify the process of copying them into your forensic log.
Video Demonstration
The following video describes how to use the script:
https://www.youtube.com/watch?v=j8gaQz9fezQ
External Links
ReadMe file
The sleuth kit howto -
To find which file maps to a given sector number on the first partition of a Windows volume:
ifind -o63 -d
This returns an inode number.
ffind -o63
For example, if we want to find which file maps to sector 1249 of the file image.iso, use:
$ ifind -o63 -d 1249 image.iso 6 $ ffind -o63 image.iso 6 /TCLAIM.TXT
You can print out the contents of the file with the icat command:
$ icat -o63 image.iso 6 �����������������������������������������������������������������������������ͻ � ������������ ����������� �� �� �������� ��� ��� � � �� �� �� �� ���� �� ���� ���� � � �� �� �� �� �� �� �� �� �� �� � � �� �� �� �� �� �� �� ��� �� � � �� �� �� �� �� �� �� �� � � �� �� �� �� �� �� �� �� � � �� �� �� ���������� �� �� �� � � �� �� �� �� �� �� �� �� �� � � �� ����������� ����������� �� �� �������� �� �� � � � �����������������������������������������������������������������������������ͼ
Ntop -
Overview
ntop is a network traffic probe that shows the network usage, similar to what the popular top Unix command does. ntop is based on libpcap and it has been written in a portable way in order to virtually run on every Unix platform and on Win32 as well.
ntop users can use a a web browser to navigate through ntop (that acts as a web server) traffic information and get a dump of the network status 1.
What ntop can do for me?
Sort network traffic according to many protocols
Show network traffic sorted according to various criteria
Display traffic statistics
Store on disk persistent traffic statistics in RRD format
Identify the indentity (e.g. email address) of computer users
Passively (i.e. without sending probe packets) identify the host OS
Show IP traffic distribution among the various protocols
Analyse IP traffic and sort it according to the source/destination
Display IP Traffic Subnet matrix (who’s talking to who?)
Report IP protocol usage sorted by protocol type
Act as a NetFlow/sFlowcollector for flows generated by routers (e.g. Cisco and Juniper) or switches (e.g. Foundry Networks)
Produce RMON-like network traffic statistics
Platforms
Unix (including Linux, *BSD, Solaris, and MacOSX)
Win32 (Win95 and above including Vista
Loopback
Ethernet (including 802.11Q)
Token Ring
PPP/PPPoE
Raw IP
FDDI
FibreChannel
...and many more
Requirements
Memory Usage
It depends on the ntop configuration, number of hosts, and number of active TCP sessions. In general it ranges from a few MB (little LAN) to 100 MB for a WAN.
CPU Usage
It depends on the ntop configuration, and traffic conditions. On a modern PC and large LAN, it is less than 10% of overall CPU load.
Protocols
IPv4/IPv6
IPX
DecNet
AppleTalk
Netbios
OSI
DLC
…and many more
IP Protocols
Fully User Configurable
Additional Features
VoIP support (SIP, Cisco SCCP and Asterisk IAX)
NetFlow (including v5 and v9) and IPFIX support
Network Flows
Local Traffic Analysis
Multithread and MP (MultiProcessor) support on both Unix and Win32
Python lightweight API for extending ntop via scripts
Support of both NetFlow andsFlowas flow collector. ntop can collect simultaneously from multiple probes.
Traffic statistics are saved into RRDdatabases for long-run traffic analysis.
Internet Domain, AS (Autonomous Systems), VLAN (Virtual LAN) Statistics
Network assets discovery and categorization according to their OS and users
Protocol decoders for many internet protocols
Advanced ‘per user’ HTTP password protection with encrypted passwords
RRDsupport for persistently storing per-host traffic information
Passive remote host fingerprint (Courtesy of ettercap)
HTTPS (Secure HTTP via OpenSSL)
Virtual/multiple network interfaces support
Graphical ntop launcher (Win32 only)
External Links
Get ntop
Sources
ntop.org
Apple file system (apfs) -
APFS, or Apple File System, is the file system designed by Apple to supersede HFS+ and take advantage of flash/SSD storage and native encryption support. APFS also introduced file system snapshots, support for sparse files, and greater time stamp granularity.
It was announced at the Apple Worldwide Developer Conference (WWDC) in 2016 and is meant to be used with watchOS, iOS, tvOS, and macOS. A Developer Preview version was first released with macOS Sierra 10.12. APFS became the default file system with the release of macOS High Sierra 10.13 released to the public on September 25, 2017.
Features
Max file size of 263 bytes was mantinted from HFS+ and new features include:
Snapshots (can be mounted read-only)
Atomic Safe-save (single transaction save)
File and directory clones (without using additional storage space)
Space-Sharing (volumes grow and shrink, sharing underlying free space)
Sparse file support (more efficient empty space representation)
Fast directory sizing (more efficient total space computation of a directory)
Feature 	Mac OS Extended (HFS+) 	Apple File System (APFS) 	
Number of allocation blocks (Maximum Number of Files) 	232 (4 billion) 	263 (9 quintillion) 	
File IDs (iNode numbering) 	32-bit 	64-bit 	
Time stamp granularity 	1 second 	1 nanosecond 	
Crash protection 	Journaled 	Copy-on-write 	
Full disk encryption 	Yes (FileVault) 	Yes (native) 	
Extended Attribute Support 	Yes (retrofitted) 	Yes (native) 	
External Links
Apple File System Guide - Introduction
Introducing Apple File System
Slides from filesystem talk at MacSysAdmin 2016 by Rich Trouton
Creating APFS volumes by Howard Oakley
Using message id headers to determine if an email has been forged -
According to the RFCs for email, RFC 822 and RFC 2822, every email should have a message id field that: "provides a unique message identifier that refers to a particular version of a particular message. The uniqueness of the message identifier is guaranteed by the host that generates it. ... This message identifier is intended to be machine readable and not necessarily meaningful to humans. A message identifier pertains to exactly one instantiation of a particular message; subsequent revisions to the message each receive new message identifiers."
The message id headers can prove useful when trying to determine if a email is authentic. Although they can't always prove that message is authentic, they can often show that a message has been forged.
Repeated Message ID
In this case, the forger, when creating a fake email, reuses the headers belonging to an earlier message. The examiner need only compare the Message-ID from the email in question to all of the other email messages in the world. Ok, probably not all of the other email messages out there. Usually just the messages on the systems in question are good enough. But finding the same message id on the "smoking gun" email and an old guacamole recipe can be used as evidence that a message was forged.
Impossible Message ID
This case is more subtle, but can be used quite effectively. Although the RFC states that the message id should be globally unique, it says nothing about how it should be constructed. Most email programs have their own format for generating the message id. For example, Apple Mail uses a Universally Unique Identifier and the sender's domain. Thunderbird, on the other hand, uses a combination of the time the message was sent, a salt, and the sender's domain.
Sample Apple Mail Message ID:
38D1C1FD-3C35-4568-925C-FC46CAC0DE8A@sendinghost.com 
Sample Thunderbird Message ID:
41B5F981.5040504@sendinghost.com 
If a message was purportedly sent by a certain email program but does not have a message id created by that program, it has obviously been forged. It would be like a round cookie-cutter making square holes; it just can't happen. Thus, if somebody claims that they received an authentic email, look at the message-id and the format of the headers. If the message id does not match the format for that program, the message has been forged!
See Also
Email Headers
Using signature headers to determine if an email has been forged
Rekall -
Rekall is the stand-alone continuation of the Volatility version, aka the scudette branch.
One of Rekalls goals is to provide better integration with GRR by improved modularity of the framework and having memory acquisition capability.
Memory acquisition drivers
The drivers can be found under:
rekall/tools/linux rekall/tools/osx rekall/tools/windows 
Linux
lmap
In rekall RC11 the advanced Linux acquisition tool (LMAP) was added. lmap allows to inject the pmem functionality into existing kernel modules to bypass having to build a pmem kernel module for every different kernel version. See the corresponding DFRWS EU 2014 paper for more information about LMAP. LMAP is also known as the "pmem parasite".
pmem
To build the kernel module for the current kernel version, make sure you have a working build environment and the kernel headers installed. Change into this directory and run make:
cd rekall/tools/linux/ make 
The acquisition driver is named pmem.ko.
To load the driver:
sudo insmod pmem.ko 
To check if the driver is running:
sudo lsmod 
The driver create a device file named:
/dev/pmem 
To unload the driver:
sudo rmmod pmem 
To read acquire the memory just read from the device file. e.g.
dd if=/dev/pmem of=image.raw 
For more information see:
rekall/tools/linux/README 
Mac OS X
For more information see:
rekall/tools/osx/OSXPMem/README 
Windows
Since recent versions of Windows require a signed driver rekall comes with both pre-built (signed binary) and source versions of the driver.
Both the i386 and amd64 binary version of the driver can be found in the directory:
rekall/tools/windows/winpmem/binaries 
E.g.
rekall/tools/winpmem/binaries/amd64/winpmem.sys 
A standalone tool for imaging memory that uses an embedded copy of the pmem driver can be found as winpmem.exe in:
rekall/tools/winpmem/executables/Release/ 
To load the driver:
winpmem.exe -l 
The device filename is (This can not be changed without recompiling):
\\.\pmem 
Note that running dd directly on this device file can crash the machine. Use the winpmem.exe tool instead because it handles protected memory regions.
To read and acquire the physical memory and write it to image.raw:
winpmem.exe image.raw 
To unload the driver:
winpmem.exe -u 
For more information see:
rekall/tools/windows/README 
See Also
Memory analysis
Memory Imaging
Volatility
External Links
Project site
Rekall Memory Forensics blog
Using signature headers to determine if an email has been forged -
Email signatures, designed for authentication, non-repudiation, and spam control, can also be used to determine if an email has been forged. In the most common case, the forger copies headers from an existing email message to a new one. He could also attempt to change the content of a signed message. Regardless, the signature no longer corresponds to the message and it can be shown that the message is not authentic.
Sender Policy Framework
Sender Policy Framework, or SPF records, can be used to enumerate which hosts are allowed to send mail for that domain. If a message purports to be from example.com but came from an IP address that is not authorized, under SPF, to send mail for that domain, it has been forged. As an example, let's say we have an email purporting to be from lexluthor@mit.edu using the IP address 4.61.91.6. An investigator can query the DNS records from that domain, in this case mit.edu to see the SPF senders.
$ host -t txt mit.edu mit.edu descriptive text "v=spf1 ip4:18.7.7.0/24 ip4:18.7.21.0/24 ip4:18.72.0.0/16 ~all" 
Here we see that there are three computers authorized to send mail for the mit.edu domain. None of them have the IP address found on the message, so the email in question may be forged.
DomainKeys Identified Mail
These headers, included by the mail server, provide a signature of each message. See Gmail Header Format. The public keys are distributed via DNS.
Signed mail
Some other programs can be used by the sender to sign an email message. Programs such as PGP and GnuPG.
PGP Messages
Messages sent using PGP, or its free equivalents such as GnuPG, have the signature in the message body itself. Each message can be signed, encrypted, or both. Encrypted messages begin with the header
-----BEGIN PGP MESSAGE----- 
followed by some optional headers. The optional headers may include the character set of the decoded message, the program and version that created the message, and an optional comment. The end of the message is noted with
-----END PGP MESSAGE----- 
Between these two lines are a series of ASCII characters that represent the encrypted or signed message.
A signed message has the header
-----BEGIN PGP SIGNATURE----- 
at the end of the signed message followed by the same optional headers as encrypted messages. The signature is usually three lines of ASCII characters.
See Also
Using message id headers to determine if an email has been forged
Hashkeeper -
Run by the National Drug Intelligence Center, part of the U.S. Department of Justice.
HashKeeper is a database application of value primarily to those conducting forensic examinations of computers on a somewhat regular basis.
Overview
The application uses the MD5 file signature algorithm to establish unique numeric identifiers (hash values) for known files and compares those known hash values against the hash values of Computer file|files on a seized computer system. Where those values match, the examiner can say, with statistical certainty, that the corresponding files on the seized system have been authenticated and therefore do not need to be examined.
Origins
Created by the National Drug Intelligence Center (NDIC)—an agency of the United States Department of Justice—in 1996, it was the first source for hash values of "known to be good" files.
Availability
HashKeeper is available, free-of-charge, to law enforcement, military and other government agencies throughout the world. It is available to the public by sending a Freedom of Information Act request to NDIC.
External Links
Official NDIC website
Security onion -
Description
Security Onion is a distribution of Linux which comes with several forensic, IDS, and NSM tools pre-installed. Although Security Onion is mainly intended for IDS and NSM, it does provide a useful platform for performing forensics, as it comes with many forensics tools installed.
The image can be downloaded at: https://blog.securityonion.net/p/securityonion.html
List of Installed Tools
The following is the list of tools included on the Security Onion distro, as listed at https://github.com/security-onion-solutions/security-onion:
abcip
argus
barnyard2
bittwist
Bro
chaosreader
Daemonlogger
driftnet
dsniff
Dumbpig
ELSA
fwsnort
Hogger
hping
httpry
hunt
inundator
labrea
mergecap
ncat
netsed
netsniff-ng
NetworkMiner
nftracker
ngrep
nmap
oinkmaster
OSSEC
ostinato
p0f
pcapcat
ptunnel
Reassembler
scapy
sguil
Sniffit
Snorby
Snort
SnortValidator
Squert
ssldump
sslsniff
Suricata
tcpdump
tcpick
tcpreplay
tcpslice
tcpstat
tcpxtract
traceroute-circl
tshark
u2boat
u2spewfoo
udptunnel
Vortex
Wireshark
xpipes
Xplico
xprobe2
Zenmap
External Links
Official Website
Md5 -
The Message-Digest algorithm 5 (MD5) is a cryptographic hash function that produces a 128-bit hash value. Originally developed in 1991, much as has been written about this algorithm. As such, this article concentrates only on its application to computer forensics.
Tools
On most Unix systems the tools digest, md5 (BSD) or md5sum can be used to compute the MD5 hash of a file or device. md5deep can compute MD5 hashes of whole directory trees.
Weaknesses
Recently some cryptographic weaknesses have been found in MD5. Tool developers should avoid using MD5 in new products in favor of other hash functions like RIPEMD-160, Tiger, WHIRLPOOL, SHA-256 or SHA-512. Host Intrusion Detection systems and hash databases should also use multiple hash algorithms.
External Links
Wikipedia: MD5
Collection of exploits and weaknesses in MD5
Ffs -
FFS is one name given to the BSD Fast File System, also known as UFS1 or UFS2 (referring to different versions of the file system). Modern FFS implementations are used in many systems based on 4.2BSD, 4.3BSD and 4.4BSD, including FreeBSD, OpenBSD, and NetBSD. Variants and derivative file systems are also in use on other platforms, including Solaris. Many differences in terms of capabilities and metadata exist between these systems, and compatibility is not guaranteed.
Note: This article refers to the BSD Fast File System. For information on the Amiga Fast File System, please see FFS Amiga.
More Information
More information regarding super-block metadata structures can be gleaned from the fs(5) manual page in FreeBSD.
An email list, freebsd-fs, exists and can be viewed online via Google Groups.
Wikipedia's article.
Selective file dumper -
Selective File Dumper (SFDumper) is a tool written in Bash Script for Linux systems.
It's fast and selective, it can retrieve all the files of the file type you choose with only one tool referenced, deleted and unallocated in very fast way.
The Bash script SFDUMPER.SH can recover active, deleted and unallocated files automatically and then it can delete the carved duplicate files of the deleted and active files retrieved by the The Sleuth Kit, thanks to the comparison of the SHA256 hash codes.
It's possible to recognize the renamed files by the data carving and it's possible to expand the Foremost configuration file inside the script, for adding new extensions.
Finally, it is possible to do a keywords search on the set of files extracted by the The Sleuth Kit and Foremost.
The script can work on the partition chosen from an image file or directly from the device (eg. /dev/sdb).
Actions
1) Choosing the partition to analyze from an image file or a device; 2) Choosing the file type by the extension you need to have; 3) Extracting all referenced files by their extension; 4) Extracting all the deleted files by their extension; 5) Carving all the partitions chosen and, automatically, the script will delete the duplicate files leaving only the carved files whose are not into the referenced or delete set of files; 6) Executing a keyword search on all the retrieved files; 7) Reporting all with the investigator name, date and time.
Requirements
Linux
The Sleuth Kit
Foremost
md5deep
sed, grep and awk
dd
Requirements for the GUI version
Zenity
Usage
sudo sh sfdumper.sh
or
chmod +x sfdumper.sh
./sfdumper.sh
Official website
https://sfdumper.sourceforge.net/