hemu1438/forensicsgpt-training-data huggingface dataset:
Mandiant -
Mandiant is a US-based cybersecurity company that was acquired by Google Cloud in 2022. Mandiant was originally founded as Red Cliff Consulting in 2004 before rebranding in 2006. Mandiant gained significant fame in February 2013 when it released the APT1 report, a report detailing and implicating China's efforts, specifically China's 2nd Bureau of the People's Liberation Army (PLA) General Staff Department's (GSD) 3rd Department (Military Cover Designator 61398), in cyber espionage.
In December 2013, Mandiant was acquired by FireEye, and the combined company offered incident response, consulting, and proactive services, managed detection and response (MDR), and various cybersecurity products. The FireEye name and product line were sold to Symphony Technology Group in June 2021.
In March 2022, Google announced its plans to acquire Mandiant and integrate it in Google Cloud. The acquisition was completed in September 2022. The Mandiant brand will be retained as a part of Google Cloud.
Since its inception, Mandiant has significant contributions to the DFIR community, including (but not limited to):
Discovery and release of new forensic artifacts.
Disclosure of threat intelligence reports and threat actor TTPs.
Release and maintenance of free, open or closed source tools.
List of Mandiant Tools
Product Name Description Version Status Link 
capa	Open source tool to identify capabilities within an executable file	4.0.1	Active	Link	
Commando-VM	A Windows-based security distribution for penetration testing and red teaming	N/A	Active	Link	
First Response	An agent-based incident response tool.	N/A	Deprecated	N/A	
Flare-VM	A Windows-based security distribution for malware analysis, incident response, and other cybersecurity activities	3.0.1	Active	Link	
Floss (Flare Obfuscated String Solver)	A tool to automatically extract obfuscated strings from malware	2.1.0	Active	Link	
Red Curtain	Originally released at BlackHat Federal in 2007, this tool was described as a malware detection product	N/A	Deprecated	N/A	
Web Historian	A free tool to parse web browser history from FireFox 2/3+, Chrome 3+, and Internet Explorer versions 5 through 8	2.0	Active	Link	
External Links
Official website
National software reference library -
The National Software Reference Library (NSRL) is the National Institute of Standards and Technology's National Software Reference Library. The NSRL is a physical resource located in Gaithersburg Maryland. The NSRL consists of more than 21,000 individual software packages. NIST has the original packaging and distribution media for 15,000 packages, and archived digital distributions for the remainder.
RDS is the Reference Data Set. The RDS consists of the metadata describing software package titles, manufacturers, operating systems, file metadata and hash codes of the files in the NSRL. Originally (ca. 2000-2002) it was created by installing the software on systems and then generating a list of the hash codes. During 2003-2012 it was created by processing only the distributed media and extracting the installation files from the Microsoft .CAB, .MSI and .ZIP files, Unix/Linux .RPM and .DEB files, Apple .DMG files, and generic archive files. As of 2013, operating systems and packages are installed in virtual machines to facilitate collection of metadata and hashes from those installations.
The RDS is typically used for data reduction. That is, the set of hash codes is used as a filter to eliminate or highlight files from examination. Most frequently the RDS used as a list of known goods that can be safely suppressed. This is an incorrect usage of the RDS and should be discouraged, because the RDS does not indicate if a file is known good or bad, only that it is known. Indeed, the RDS has many files that were once throught to be good, but are now thought to be bad---for example, versions of Adobe Flash with known security vulnerabilities. Categories such as "Steganography" or "Keylogger" are assigned to allow filtering by need.
The NSRL is distributed online can be downloaded from the NSRL website.
RDS File Format
Each RDS consists of several files, but the hashes are stored in NSRLFile.txt. These files have a header followed by many hash records. The header denotes the columns in each file. (See the External Links for the complete specification). RDS files can be used directly with programs like md5deep, FTK, and EnCase.
The file format has changed slightly over time. Releases occur four times per year. The latest version was dated 1 Mar 2013:
Version 2.0
Starting in version 2.0, the NSRL moved the hashes to the start of each line and dropped the MD4 hash. The file header:
"SHA-1","MD5","CRC32","FileName","FileSize","ProductCode","OpSystemCode","SpecialCode" 
Version 1.5
Information on the older header version is kept here so that programs can read older files. The file header:
"SHA-1","FileName","FileSize","ProductCode","OpSystemCode","MD4","MD5","CRC32","SpecialCode" 
OpSystemCode refers to the operating system code. The SpecialCode is a single character that can be used to mark records. A normal file has a blank value here. An M in this field denotes a malicious file.
External Links
NSRL website
Data Formats of the NSRL Reference Data Set (RDS) Distribution
Identifying file systems -
There are many ways to identify a file system inside a partition.
When an operating system attempts to mount a partition, it may use the ID from the partition table.
There are several tools that can identify the partition from the data it contains, including:
blkid
file
parted
testdisk
In general you should not rely on the partition record for determining the partition type.
Some of the test images created for the Honeynet Challenges have multiple file systems contained in a single partition.
Tcpdstat -
Overview
Tcpdstat is an old, but useful tool for analyzing network based evidence. Tcpdstat offers very helpful statistical information about packet captures, including start/finish timestamps (to identify the total duration of the packet capture), breakdown by protocol/port, and packet size distribution.
Tcpdstat is useful for analyzing network trace data because it gives a high-level overview, which can identify where to look next. For example, if quite a few packets show up for a protocol/port that isn’t normally used on the victim machine, you've already been clued in on what to look for next on a lower level.
External Links
Unofficial fork
Sha 1 -
SHA-1 (Secure Hash Algorithm) is a cryptographic hash function that was first published in 1995 in Federal Information Processing Standards (FIPS) PUB 180-1 by the National Security Agency (NSA) as a fix to an unknown flaw found in SHA-0.
In February of 2005 an attack was announced that could find collisions in SHA-1 requiring fewer than 269 operations.
Tools
On most Unix systems the tool sha1sum can be used to compute the SHA-1 hash of a file or devices.
External Links
RFC 3174
The Hacker Defense (aka Trojan/Virus Defense)
A grab-bag collection of citations on legal issues.
Below are accounts of different hacker/virus/Trojan related defenses. Albeit some of these are not ‘reputable’ web sources, but they should all have official court backing from wherever the various investigators that do similar. The Crown Prosecution Service (covers England and Wales) is training prosecutors en masse about ‘trojan defenses’ (link below). These types of actions would not occur unwarranted. Why do all the extra work for nothing?
“The "Trojan defense" has now become standard in many types of computer crime cases. But the defense often plays on the ignorance of juries and prosecutors. It has raised the need for the CPS to do more to explain complex technical issues in simple terms to judges and juries, says George.” (Esther George is the policy adviser at the Crown Protection Services) 1
US man, Eugene Pitts, found not-guilty of tax evasion after blaming a computer virus. Avoids \~\$900,000 in fines.
Karl Schofield walked free from court yesterday after prosecutors accepted an expert's report that the "Trojan" program could have saved the 14 depraved images off the internet without his knowledge.
Julian Green, 45, of Torquay, Devon was cleared in court in July of 13 charges of making indecent images, claiming computer malware was to blame.
A former Georgia teacher blames computer viruses for altering his Web sites and uploading child porn images. Guilty charge upheld.
Odd spin on the issue, where a hacker used a Trojan to gain access to potential pedophile’s computers.
Bandy’s defense attorney asserted that a “virus” or “trojan” must have downloaded the child pornography to Bandy’s computer without his knowledge. 4
A man found with more than 1,700 indecent images of children on his computer claimed a virus was to blame, a court heard. But Mark Craney, 33, from Knowle, was found guilty at Warwick Crown Court on 16 charges of making indecent images of children by downloading them onto his computer. 5
More links from previous research.
6
7
8
Connecticut v. Amero
From Wikipedia:
The defendant in the case, Julie Amero (born 1967), a substitute teacher, was previously convicted of four counts of risk of injury to a minor, or impairing the morals of a child, as the result of a computer that was infected with spyware and DNS hijacking software; the conviction was vacated on appeal.
Wikipedia: Connecticut v. Amero
Technical review of the Trial Testimony State of Connecticut vs. Julie Amero, March 21, 2007
External Links
THE TROJAN HORSE DEFENSE IN CYBERCRIME CASES, by Susan W. Brenner, Brian Carrier, and Jef Henninger, in 2004
The "Tools Proven in Court" Question
Privacy and Surveillance Laws
18 USC 2510 et seq., 18 USC 2701 et. seq., 18 USC 1030 and other statutes regulate the information private entities and law enforcement can access over a computer network.
Cybersecurity Research
Toward a Culture of Cybersecurity Research, Aaron J. Burstein, University of California, Berkeley - School of Law. 2008, UC Berkeley Public Law Research Paper No. 1113014
Volatility framework -
The Volatility Framework is a completely open collection of tools, implemented in Python under the GNU General Public License (GPL v2), for the extraction of digital artifacts from volatile memory (RAM) samples. The extraction techniques are performed completely independent of the system being investigated but offer unprecedented visibility into the runtime state of the system. The framework is intended to introduce people to the techniques and complexities associated with extracting digital artifacts from volatile memory samples and provide a platform for further work into this exciting area of research.
The project was originally developed by and is now headed up by AAron Walters of Volatile Systems.
Plugins
See: List of Volatility Plugins
Memory acquisition drivers
In 2012 Michael Cohen contributed both a Linux and a Windows Open Source memory (acquisition) driver to the Volatility project as part of the Technology Preview (TP) version, aka scudette branch. Since the scudette branch of Volatility has moved on as a separate project, the drivers can now be found as part of the rekall project.
See Also
List of Volatility Plugins
External Links
Official website
Code repository, direct link to source
Volatility Documentation
Third party documentation
Set Up to More Memory Forensics!, October 2011
Memory Forensics With Volatility (Technology Preview), by Michael Cohen, October 2012
Using Volatility: Suspicious Process (1/2)
Using Volatility: Suspicious Process (2/2)
Zeitgeist -
Zeitgeist is a service which logs the users's activities and events, anywhere from files opened to websites visited and conversations.
activity.sqlite
$HOME/.local/share/zeitgeist/activity.sqlite 
event_view seems one of the more interesting tables (which is technically a view and not a table).
The timestamp is formatter in milli seconds sinds POSIX epoch (Jan 1, 1970 00:00:00 UTC)
SELECT datetime((timestamp/1000), "unixepoch"), subj_uri, subj_origin_uri, subj_text, subj_storage, event_origin_uri, actor_uri FROM event_view; 
External Links
Wikipedia: Zeitgeist (framework)
Alt linux rescue -
ALT Linux Rescue is yet another sysadmin's Live CD with some forensic capabilities and features.
Intro
This weekly-updated image is intended to be text-only toolchest for data analysis and recovery.
It will not try to use swap partitions or autodetect and automount file systems unless requested explicitly.
Forensic mode is available via a separate boot target for BIOS users and a rescue boot option (via F2) for UEFI users. This will skip activating MDRAID/LVM too.
Build profile suitable for ALT Linux mkimage tool is included as .disk/profile.tgz.
Tools included
Most of the usual rescue suspects should be there, such as: biew, chntpw, dc3dd/dcfldd, foremost, john, md5deep, nmap, scalpel, The Sleuth Kit, ssdeep, wipefreespace
to name a few are available either; libevt, libevtx, liblnk, libpff, libregf, libuna, libvshadow, libwrc tools have been added since 20140514.
X11-based software is being considered for an extended version.
Platforms
i586 (BIOS) and x86_64 (BIOS/UEFI); SecureBoot might be left enabled in most occasions (Hyper-V gen.2 is a notable exception, looks like Microsoft's "3rd party driver" key is lacking there; just turn SB off).
Deliverables
Two separate 32/64-bit hybrid ISO images suitable for direct writing onto USB Flash media (or CD-R by chance).
Forensic issues
Hardening against rootfs spoofing has been implemented as of 20140423 (stage2 squashfs SHA256 check has been contributed by Maxim Suhanov); previous images are vulnerable to ISO9660-on-device containing a squashfs file with predefined name and specially crafted contents.
MDRAID/LVM2/swaps activation might occur with images before 20140416 or when booted via the default "Rescue" target; booting into "Forensic mode" will skip that (for both early userspace and final environment as of 20140416) and switch mount-system script to use ro,loop,noexec mount options (as of 20140423).
Physical device write blocking hasn't been considered so far.
External Links
Project site (also available in Russian)
Part of Regular Builds based on ALT Linux Sisyphus
Rescue image within ALT Linux Starterkits based on stable branch has gained the same features as of 20140612
Knoppix std -
Knoppix security tools distribution (STD) is a computer forensics and incident response Live CD based on Knoppix.
Tools
Forensics
The Sleuth Kit 1.66 : extensions to The Coroner's Toolkit forensic toolbox.
autopsy 1.75 : Web front-end to TASK. Evidence Locker defaults to /mnt/evidence
biew : binary viewer
bsed : binary stream editor
consh : logged shell (from F.I.R.E.)
coreography : analyze core files
dcfldd : US DoD Computer Forensics Lab version of dd
fenris : code debugging, tracing, decompiling, reverse engineering tool
fatback : Undelete FAT files
foremost : recover specific file types from disk images (like all JPG files)
ftimes : system baseline tool (be proactive)
galleta : recover Internet Explorer cookies
hashdig : dig through hash databases
hdb : java decompiler
mac-robber : TCT's graverobber written in C
md5deep : run md5 against multiple files/directories
memfetch : force a memory dump
pasco : browse IE index.dat
photorec : grab files from digital cameras
readdbx : convert Outlook Express .dbx files to mbox format
readoe : convert entire Outlook Express .directory to mbox format
rifiuti : browse Windows Recycle Bin INFO2 files
secure_delete : securely delete files, swap, memory....
testdisk : test and recover lost partitions
wipe : wipe a partition securely. good for prep'ing a partition for dd
and other typical system tools used for forensics (dd, lsof, strings, grep, etc.)
External Links
Official Site
Winfe -
Windows Forensic Environment - a forensically sound bootable CD/USB to acquire electronic media or conduct forensic analysis.
Windows Forensic Environment ("WinFE")
WinFE was developed and researched in 2008 by Troy Larson, Sr Forensic Examiner and Research at Microsoft. WinFE is based off the Windows Pre-installation Environment of media being Read Only by default. It works similar to Linux forensic CDs that are configured not to mount media upon booting. However, unlike Linux boot CDs, with Win FE one can use Windows based software. Thus it is possible to include various forensic software and general portable utilities. WinFE can also be configured to boot from a USB device, should the evidence computer have the ability to boot to USB.
WinFE can be customized to the examiner's needs through batch files using the Windows Automated Install Kit (WAIK) or through 3rd party utilities such as WinBuilder 2.
Some examples of Windows based forensic utilities that can run in the Windows Forensic Environment include:
AccessData FTK Imager,
Guidance Software Encase,
ProDiscover
RegRipper.
X-Ways Forensics,
A write protection tool developed by Colin Ramsden was released in 2012 that provides a GUI for disk toggling Colin Ramsden's write protect tool effectively replaces the command line to toggle disks on/offline or readonly/readwrite.
Technical Background and Forensic Soundness
Windows FE is based on the modification of just two entries in the Windows Registry. The first key is located at "HKEY_LOCAL_MACHINE\system\ControlSet001\Services\MountMgr". The DWord "NoAutoMount" has to be set to "1". By doing this the Mount-Manager service will not automatically mount any storage device. The second key is "HKEY_LOCAL_MACHINE\system\ControlSet001\Services\partmgr\Parameters" where "SanPolicy" has to be set to "3". While both keys will avoid the mounting of storage devices the user has to mount the storage drive manually by using the command-line tool DiskPart, while the evidence drive does not need to be mounted for imaging/forensic access.
The latest modification (New for Windows 8) to the registry is SAN policy 4. SAN policy 4 Makes internal disks offline. Note. All external disks and the boot disk are online.
Testing has shown that mounting a volume in READ ONLY mode will write a controlling code to the disk, whereas mounting a disk in READ ONLY mode will not make any changes. Depending on the type of filesystem there is a potential modification to the disk with a documented 4-byte change to non-user created data. This modification exists for non-Windows OS disks, where Windows (FE) will write a Windows drive signature to the disk, although it is not shown to be consistent. Various issues with Linux Boot CDs can be compared 1 ).
Resources:
Windows Forensic Environment blog
Article on Win FE in Hakin9 magazine 2009-06
WinPE Technical Reference
Windows Automated Installation Kit
Malware -
Malware is a short version of Malicious Software.
Malware is software used for data theft, device damage, harassment, etc. It is very similar to computer malware. It installs things such as trojans, worms, and botnets to the affected device. It is illegal to knowingly distribute malware.
Virus
A computer program that can automatically copy itself and infect a computer.
Worm
A self-replicating computer program that can automatically infect computers on a network.
Trojan horse
A computer program which appears to perform a certain action, but actually performs many different forms of codes.
Spyware
A computer program that can automatically intercept or take partial control over the user's interaction.
Exploit Kit
A toolkit that automates the exploitation of client-side vulnerabilities, targeting browsers and programs that a website can invoke through the browser 1. Often utilizing a drive-by-download.
Drive-by-download
Any download that happens without a person's knowledge 2.
Rootkit
A rootkit is a stealthy type of software, typically malicious, designed to hide the existence of certain processes or programs from normal methods of detection and enable continued privileged access to an operating system.
Various types of rootkits:
User mode
Kernel mode
Bootkits
Hypervisor level
Firmware and Hardware
See Also
Malware analysis
Windows Registry Persistence keys
External Links
Wikipedia: malware
Wikipedia: drive-by-download
Androguard: A list of recognized Android malware
Communities @ Risk - Targeted Digital Threats Against Civil Society
Vulnerability Disclosure in the Age of Social Media: Exploiting Twitter for Predicting Real-World Exploits, by Carl Sabottke, Octavian Suciu, Tudor Dumitras, Usenix 2015
Analysis
A Forensic Overview of a Linux perlbot, by Andre M. DiMino, December 17, 2013
Probing into the Flash Zero Day Exploit (CVE-2014-0502), by Krishnan Subramanian, February 21, 2014
Operation Windigo, by Olivier Bilodeau, Pierre-Marc Bureau, Joan Calvet, Alexis Dorais-Joncas, Marc-Étienne M.Léveillé, Benjamin Vanheuverzwijn, March, 2014
Security Advisory 2953095: recommendation to stay protected and for detections, by Chengyun Chu, Elia Florio, March 24, 2014
Exploit Kit
What Are Exploit Kits?, by Lenny Zeltser, October 26, 2010
Exploit Kits 2014-2015
Persistence
Many ways of malware persistence (that you were always afraid to ask), May 5, 2015
Rootkit
Wikipedia: Rootkit
Understanding Rootkits: Using Memory Dump Analysis for Rootkit Detection, by Dmitry Korolev, Yuri Gubanov, Oleg Afonin, November 22, 2013
Turning USB peripherals into BadUSB
Phison 2251-03 (2303) Custom Firmware & Existing Firmware Patches
Digital evidence bags -
The Digital Evidence Bag (DEB) format mimics in a digital environment the bags, tags and seals used to traditionally wrap evidence.
When a DEB is created three files are generated:
A .tag file which is plain text and stores case specific metadata such and evidence reference identifier, examiner, location, timestamps and tag continuity blocks that record DEB access activity. In addition to this the tag file contains the cryptographic hashes that are used to maintain and assure the integrity of the DEB structure.
An .index file is a plain text file that records device, file or data source metadata.
A .bag file that holds the evidential data e.g. the raw device bit stream, logical files, network packet capture data.
Eml -
EML or Electronic mail is a file format primarily associated with Microsoft Corporation “Outlook Express” and is used by other email clients as well. It is a file extension (.eml) for e-mail message saved to file in MIME RFC 822 standard format. Basic purpose of EML file is storing email messages as a plain text file in standard file structure. It can contain plain ASCII text for headers, main message body, attachments and hyperlinks as well. Header comprises of e-mail address of sender and recipient, subject, time & date stamp of message.
Overview
EML file extension for email files also referred as RFC-822 file format was created in 1982 as an enhancement in RFC-733 through establishment of ARPA standards for emerging use of emails. Its purpose was also to simplify the address standards which were not uniformly recognized in software business.
Microsoft Corporation also manufactured series of COM modules which was included in Outlook Express email application’s development as well. These modules were part of Microsoft Office suite products. RFC is a standard format but Microsoft create PST file format to store emails as a highly structured database. But it restricted the access of emails without presence of proprietary application module. This file format was created to store email files offline with a functionality of forwarding email messages. Eventually this created an issue for non-Microsoft email clients when received from Outlook application. Since then, many other email applications created emails as .eml files saving the file in MIME RFC-822 format. Primarily associated with Outlook Express EML file format is also part of other email clients like Thunderbird, Apple Mail, etc. Outlook Express successor email client Windows Live Mail also use EML file format to save emails.
MIME Type
message/rfc822
Support
Being a plaintext and analogous to MHT file format, EML can be opened in a webbrowser by changing the extension to .mht. However, Windows 8 built-in Mail application does not support EML.
Also see
MBox
Tools
Free EML Viewer
Aid4Mail
Apple Mail
Emailchemy
EML Viewer
Entourage
IncrediMail
Microsoft Outlook
Mozilla Thunderbird
Xena