hemu1438/forensicsgpt-training-data huggingface dataset:
Exiftool -
Exiftool is a Perl library and a command-line tool that can be used for reading and writing metadata in files. In addition to exif images, Exiftool also supports many other file formats, such as video and audio files as well as Word documents. Note that Exiftool does not support writing metadata to some formats. While Exiftool can be helpful in a forensic analysis, it is not a forensic tool, nor is it an anti-forensic tool.
License
Exiftool is free and open source release under the same terms as Perl.
Supported Files
The exiftool website has a comprehensive list of the supported file types, and indicates if Exiftool supports reading, writing, and/or creating metadata for each file type. Exiftool's specialty is image file formats. It can extract exif and gps metadata and it can extract Maker Notes from several popular manufacturers.
External Links
Official website
Cross drive analysis -
Cross-Drive Analysis is a computer forensic technique that correlates information found on multiple hard drives. The technique, which is still being researched, can be used for identifying social networks and for performing anomaly detection.
References
Garfinkel, S., "Forensic Feature Extraction and Cross-Drive Analysis," The 6th Annual Digital Forensic Research Workshop Lafayette, Indiana, August 14-16, 2006. https://simson.net/clips/academic/2006.DFRWS.pdf
NSF Award 0730389, EXP-SA: Prediction and Detection of Network Membership through Automated Hard Drive Analysis. https://www.nsf.gov/awardsearch/showAward?AWD_ID=0730389
Exif -
The Exchangeable image file format (Exif) is an image file format which adds metadata to existing image formats, such as JPEG.
To read the Date/Time tag do:
exif --tag=0x132 filename.jpg (reveals Date & Tag) exif --tag=0x9003 filename.jpg (reveals Date and Time (original)) exif --tag=0x9004 filename.jpg (reveals Date and Time (digitized))
To set the Date/Time tag:
exif --tag=0x132 --set-value="1975:01:01 12:01:01" --ifd=0 filename.jpg
For tools that extract Exif meta data look here - Document_Metadata_Extraction#Images
External Links
Exif 2.2 specification
Wikipedia: Exif
Converting binary plists -
Binary plists are the files that Apple products use to store information. The easiest way to view them is to convert them to XML. Note that binary data in XML plist are converted in base64 and therefore strings in the binary data are not directly visible.
The program plutil is native to Mac OS X (as of 10.2), however it is also included when iTunes is installed on a Windows PC. On linux libplist 1 also provides an equivalent of plutil.
plutil on a Windows PC is stored in:
C:\Program Files (x86)\Common Files\Apple\Apple Application Support 
Which can be added to the PATH in Environmental variables so plutil can be run from anywhere If you could like to copy out the plutil executable to another directory it requires a number of DLL's to function. These dlls are: ASL, CFNetwork, CoreFoundation, Foundation, icudt46, libdispatch, libicuin, libicuuc, libtidy, libxml2, objc, ptheadVC2, SQLite3 and zlib. These are all found in the Apple Application Support folder along with plutil.exe.
To convert Binary plists to XML run the command:
plutil -convert xml1 file.plist 
Or with libplist
plutil -i file.plist > file.xml.plist 
External Links
A library to handle Apple Property List format in binary or XML
Extended file system (ext) -
The Extended File System (Ext) is a family of file systems consisting of: ext, ext2, ext3, ext4
Ext2
ext2 or the second extended file system is a Linux file system designed as a replacement for ext.
Ext3
ext3 or the third extended file system is a Linux file system. Its main advantage over ext2 is journalling which improves reliability and eliminates the need to check the file system after an unclean shutdown.
Note that ext3 is mostly compatible with ext2.
Ext4
ext4 or the fourth extended file system is a Linux file system.
External Links
Ext2
Wikipedia: ext2
Layout of the ext2 Filesystem
Linux Ext2fs Undeletion mini-HOWTO
Ext2 Filesystem for Linux and Solaris, by Cmihai, November 3, 2007
Ext3
Wikipedia: ext3
Tools
Tools that can be used to perform recovery of data from an ext2 file system.
The Sleuth Kit
R-Studio
Hard drive -
A hard disk drive (HDD) is a non-volatile storage device which stores digitally encoded data on rapidly rotating platters with magnetic surfaces.
Data Recovery
Recovering bad data
Recovering Overwritten Data
See Also
Full Disk Encryption
Hard Drive Passwords
DCO and HPA
Advanced Format - the move to 4K sectors, and problems with Windows XP
Links
My Dead Hard Drive story or How I restored my save games
Hard drive passwords -
Some hard drives support passwords. These passwords can be implemented in computer's operating system, its BIOS, or even in the hard drive's firmware. Passwords implemented in the OS are the easiest to remove, those in the firmware are the hardest.
Sometimes people use the term "password" but the hard drive is really encrypted, and the password is used to unlock a decryption key. These passwords cannot be removed — the encryption key must be cracked or discovered through another means.
Vendors
Disklabs (www.disklabs.com) is able to remove some forms of hard drive passwords.
Dell will assist law enforcement in removing the passwords from password-protected hard drives. You need to provide Dell with a copy of the search warrant and the computer's service tag #. Reportedly this can be done over the phone, once you have a good relationship with Dell.
Seagate HDD Service Device for 2,5" drives BASIC Kit: The tool works with 2,5" drives of Seagate. Main functionality - ATA PASSWORD removal from 2,5" drives.
PC-3000 for Windows has "An opportunity to unlock USER and MASTER passwords used in a HDD".
SalvationDATA sells several products for recovery from what they call "Stage 2 physical data damage". See Data Compass.
Master Passwords
These passwords were received from unofficial sources, they may not work!
Western Digital: WDCWDCWDCWDCWDCWDCWDCWDCWDCWDCWD
Maxtor: Maxtor*INIT SECURITY TEST STEP*F (* means 00h)
Seagate: Seagate
Fujitsu, Hitachi, Toshiba: 32 spaces
Samsung: tttttttttttttttttttttttttttttttt
IBM:
CED79IJUFNATIT
VON89IJUFSUNAJ
RAM00IJUFOTSELET
Data mining -
Right now this is just a list of resources that will be useful for people doing forensic data mining and machine learning.
Open Source Software
Weka data mining toolkit - java, has programmatic and GUI interface.
Ping He has created an Open Source C4.5 implementation in C
Apache Mahout: goal is to "build scalable, Apache licensed machine learning libraries" (java). also includes a focus on using hadoop.
The Journal of Machine Learning maintains an archive of non-trivial machine learning algorithms, toolboxes, and languages.
File access conditions -
File Access Conditions
In GSM every file has File Access Condition placed on it. File Access Conditions are restrictions placed upon the file. There are 15 codes that pertain to File Access Conditions.
Level Condition Description 
0 	Always 	No Restrictions For Access 	
1 	CHV1 (Card Holder Verification 1) 	Access shall only be granted is one of the following condition is fulfilled: The correct CHV1 has been presented to the sim card, CHV1 is enabled/disabled indicator is set to disabled, or the UNBLOCK CHV1 has been successfully performed during the current session. 	
2 	CHV2 	Access shall only be granted is one of the following condition is fulfilled: The correct CHV2 has been presented to the sim card or the UNBLOCK CHV2 has been successfully performed during the current session. 	
3 	Reserved 		
4 - 14 	ADM 	Responsibility of the Administrative Authority (Conditions set by the Carrier or other Governing Bodies). 	
15 	Never 	Access can only be granted internally within the phone and can not be performed over a SIM/ME interface. 	
Warning: After 3 failed attempts at entering a CHV1 or CHV2 all access privileges granted to those numbers are lost immediately
References
GSM 11.11 Version 5.0.0 December 1995 ISBN 2-7437-0447-0
Data storage media -
There are many forms of data storage media out there, all with different properties regarding forensic investigation.
Caches
RAM, ROM, Flash
Floppy Disks
Optical media: CDs, DVDs, CD-RWs, ...
Hard drives
USB thumb drives
Memory cards, Smartcards, SIM cards, RFID tags, ...
Tapes
SSD
External Links
Wikipedia: Computer storage
Dco and hpa -
Device Configuration Overlay (DCO) and Host Protected Area (HPA).
Detection
Linux
Using hdparm
HPA
Command:
# hdparm -N /dev/sda 
Disabled HPA:
/dev/sda: max sectors = 1465149168/1465149168, HPA is disabled 
Enabled HPA:
/dev/sdc: max sectors = 586070255/586072368, HPA is enabled 
DCO
Command:
# hdparm --dco-identify /dev/sda 
Example output:
/dev/sda: DCO Revision: 0x0001 The following features can be selectively disabled via DCO: Transfer modes: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 udma6(?) Real max sectors: 1465149168 ATA command/feature sets: SMART self_test error_log security HPA 48_bit (?): selective_test conveyance_test write_read_verify (?): WRITE_UNC_EXT SATA command/feature sets: (?): NCQ SSP 
Removing HPA
Linux
Using hdparm
Command:
# hdparm -N p586072368 /dev/sdc 
(permanently (!) set max visible number of sectors, see example above)
Other Tools
TAFT (The ATA Forensics Tool), claims the ability to look at and change the HPA and DCO settings.
SAFE-Block, claims the ability to temporarily remove the HPA and remove the DCO and later return it to its original state.
HDD Capacity Restore, a reportedly Free utility that removed the DCO (to give you more storage for your hard drive!)
Tableau TD1 can remove the HPA and DCO.
External Links
Methods of discovery and exploitation of Host Protected Areas on IDE storage devices that conform to ATAPI-4, Mark Bedford, Digital Investigation, Volume 2, Issue 4, December 2005, Pages 268-275
Hidden Disk Areas: HPA and DCO, Mayank R. Gupta, Michael D. Hoeschele, Marcus K. Rogers, International Journal of Digital Evidence, Fall 2006, Volume 5, Issue 1
Removing host protected areas (HPA) in Linux, Brian Carrier, Sleuth Kit Informer #20
Wikipedia: Device Configuration Overlay
Wikipedia: Host Proteced Area
Department of justice computer crime and intellectual property section -
The Computer Crime and Intellectual Property Section of the United States Department of Justice (CCIPS) is responsible for implementing the Department's national strategies in combating computer and intellectual property crimes worldwide. The Computer Crime Initiative is a comprehensive program designed to combat electronic penetrations, data thefts, and cyberattacks on critical information systems. CCIPS prevents, investigates, and prosecutes computer crimes by working with other government agencies, the private sector, academic institutions, and foreign counterparts. Section attorneys work to improve the domestic and international infrastructure-legal, technological, and operational-to pursue network criminals most effectively. The Section's enforcement responsibilities against intellectual property crimes are similarly multi-faceted. Intellectual Property (IP) has become one of the principal U.S. economic engines, and the nation is a target of choice for thieves of material protected by copyright, trademark, or trade-secret designation. In pursuing all these goals, CCIPS attorneys regularly run complex investigations, resolve unique legal and investigative issues raised by emerging computer and telecommunications technologies; litigate cases; provide litigation support to other prosecutors; train federal, state, and local law enforcement personnel; comment on and propose legislation; and initiate and participate in international efforts to combat computer and intellectual property crime.
The Cybercrime lab is responsible for providing computer forensic and other technical support to CCIPS attorneys as it applies to implementing the Department's national strategies in combating computer and intellectual property crimes worldwide.
The Cybercrime lab supports the CCIPS comprehensive program designed to combat electronic penetrations, data thefts, and cyber attacks on critical information systems. The Cybercrime lab also provides technical support and training to improve the domestic and international infrastructure-legal, technological, and operational-to pursue network criminals most effectively. The Section's enforcement responsibilities against intellectual property crimes are similarly multi-faceted. Intellectual Property (IP) has become one of the principal U.S. economic engines, and the nation is a target of choice for thieves of material protected by copyright, trademark, or trade-secret designation.
External Links
Official website
Hashing -
Hashing is a method for reducing large inputs to a smaller fixed size output. When doing forensics, typically cryptographic hashing algorithms like MD5 and SHA-1 are used. These functions have a few properties useful to forensics. Other types of hashing, such as Context Triggered Piecewise Hashing can also be used.
Tools
There are literally hundreds of hashing programs out there, but a few related to forensics are:
md5sum - Part of the GNU coreutils suite, this program is standard on many computers.
md5deep - Computes hashes, recursively if desired, and can compare the results to known values.
ssdeep - Computes and matches Context Triggered Piecewise Hashes.
Hash Databases
National Software Reference Library The largest hash database.
Hashkeeper National Drug Intelligence Center
MD5 Reverse Hash Services
There are several online services that allow you to enter a hash code and find out what the preimage might have been. One way to find these services is to google for 'd41d8cd98f00b204e9800998ecf8427e' (the MD5 of the null string).
Here are some services that we have been able to find:
XMD5 This site is another simple MD5 reverse lookup. It claims a database with "billions" of entries. Mostly for password cracking.
Online Malware Hash Lookups
VirusTotal
Segmented Hashing
Segmented hashing produces not a single hash value for the entire image, but a list of hashes of corresponding LBA ranges of the image. By validating all hashes in a set it is still verify image integrity. Segmented Hashes are saved in a CSV file with the following format: hash, start LBA, end LBA
When Segmented hashing is useful
Segmented hashes support multi-pass imaging and handling of bad sectors: Hashes are calculated only for the imaged regions, while all bad sectors are excluded from calculation. This allows to validate a hash even when the source drive is damaged.
Better resiliency against data corruption: If an acquired image gets damaged later, regular hash is invalid upon verification making the entire image useless. With segmented hashing, only a single hash value becomes invalid, while the rest of the image can still be validated.
Seghash on GitHub is a free open-source tool for both calculating and validating segmented hashes.
Tooling
hashR build your own hash sets based on your data sources
Html -
The Hypertext Markup Language (HTML) file format is used to create/display web pages.
Its main purpose is to align text, images, or links on a website in a specific way. Web pages with .html or .htm extensions are examples of static website files. Any server or database technologies require another language on top of HTML to create dynamic features in a website. HTML files are mere plain text files whose contents follow certain rules.
HTML files are usually viewed using a web browser, but can also be opened with a variety of other programs (i.e., notepad, hex editors, etc).
HTML can trace its development from SGML as a text-based markup language. 1
XHTML
The Extensive Hypertext Markup Language (XHTML) is similar in nature to HTML, but has a stricter XML-based syntax.
External Links
Wikipedia: HTML
Wikipedia: XHTML
Wikipedia: SGML
HTML 4.01 Specification
XHTML 1.1 Specification
Iximager -
The IXimager imager application is external to the Windows operating systems.
It is based on Linux and is used in conjunction with ILook Investigator to perform the specific tasks. The program can use one of three proprietary file formats.
Use of IXimager is restricted to members of federal/state/local law enforcement or certain government agencies. The ILook analysis tool, which includes the IXimager, is available free of charge to these individuals.
See Also
IXimager file formats
External Links
Test Results for Digital Data Acquisition Tool: IXimager (Version 2.0, Feb-01 2006)
M3 data recovery -
M3 Data Recovery is one of the world's leading & professional data recovery solution and system utilities providers.
Data Recovery
M3 Data Recovery for Windows - Recover deleted files, recover data from foramtted, inaccessible, RAW, deleted or lost partition, etc. under Windows XP/Vista/7/8 and Windows Server 2003/2008/2012.
M3 Data Recovery for Mac - Recover deleted or lost files from any Mac-based hard drive as well as external hard drive, USB flash drive, SD card, memory card, etc. under Mac OSX 10.10/10.9/10.8/10.7.
Bitlocker Recovery
M3 Bitlocker Recovery - Recover deleted files from Bitlocker encrypted volume and recover data from damaged, inaccessible, corrupted, deleted or lost Bitlocker encrypted hard drive and USB flash drive.
RAW Drive Recovery
M3 RAW Drive Recovery - One step solution to fix RAW drive and then convert/change RAW File System to NTFS/FAT32 without data loss.
Partition Recovery
M3 Partition Recovery - Recover deleted partition and recover lost partition.
External Links
M3 Data Recovery Official Website
Download Link
Machine translation -
References
An evaluation of the accuracy of online translation systems
Systran Reviews - Case Studies
Spanish-to-English Translation Using the Web, Milam W. Aiken, SWDSI 2006
Me Translate Pretty One Day, Issue 14.12 - December 2006
Dispelling the myths of machine translation,By Uwe Muegge August 2008
Evaluation of a Machine Translation System for Low Resource Languages: METIS-II, Vincent Vandeghinste, etc., LREC 2008
Jhead -
jhead is a program which can display or manipulate the non-image part of Exif compliant JPEG files.
Example
This is an example of the output of jhead:
$ jhead Simpic.jpg File name : Simpic.jpg File size : 146060 bytes File date : 2006:04:04 20:10:51 Camera make : Canon Camera model : Canon PowerShot Pro1 Date/Time : 2005:12:15 14:47:02 Resolution : 779 x 524 Flash used : No Focal length : 10.6mm (35mm equivalent: 158mm) CCD width : 2.40mm Exposure time: 0.0031 s (1/320) Aperture : f/4.0 ISO equiv. : 50 Exposure bias: 1.00 Whitebalance : Manual Metering Mode: center weight Exposure Mode: Manual Exposure Mode: Auto bracketing
Macquisition boot cd -
The MacQuisition Boot CD is a forensic bootable CD/Live CD built for imaging Mac OS X systems.
Features
File Systems Understood
For Macs! Produces a dd image.
File Search Facilities
Historical Reconstruction
Can it build timelines and search by creation date?
Searching Abilities
Can it search? Does it build an index? Can it focus on file types or particular kinds of metadata?
Hash Databases
Can it create hashes of files and/or blocks? Can it compare these hash values to any databases? What sort of hash functions does it use?
Evidence Collection Features
Can it sign files? Does it keep an audit log?
History
License Notes
Is it commercial or open source? Are there other licensing options?
Linux logs analysis -
Introduction
Most Linux logs are located under /var/log/ and can be viewed using any text editor
Log files
/var/log/wtmp: Contains the logs of all logged in and logged out users. Can be analyzed using 'last' command on linux /var/log/btmp: Contains only failed login attempts. Can be analyzed using 'last' command on linux /var/log/messages or /var/log/syslog: Contains generic system activity logs /var/log/auth.log or /var/log/secure: All authentication related events in are logged here. /var/log/boot.log: Contains related information and messages logged during system startup process. /var/log/dmesg: Contains Information related to hardware devices and their drivers are logged here. /var/log/kern.log: Contains information logged by the kernel. /var/log/faillog: Contains information on failed login attempts. /var/log/cron: Contains information on cron jobs. /var/lg/yum: Contains the information that is logged when a new package is installed using the yum command. /var/log/mail.log or /var/log/maillog: All mail server related logs are stored here. /var/log/httpd/ or /var/log/apache2: Contains the logs recorded by the Apache server. Logging information are stored in two different log files – error_log and access_log. /var/log/mysqld.log or /var/log/mysql.log or /var/log/mysql/: All debug, failure and success messages related to the [mysqld] and [mysqld_safe] daemon are logged to this file. /var/log/apt/: Contains information related to Advance package tool (apt). Loggin information are stored in history.log and term.log /var/log/lastlog: Contains information about the successful logins to the host 
Resources
Dr.Ali Hadi Linux Forensics
Important Linux Log files
Mbox viewer -
MBOX Viewer is a complete application to open and view emails (along with attachments) without supporting email clients.
Features
The application thoroughly scans selected MBOX file to open its emails for preview along with attachments. Moreover, the application is a standalone and works without the support of MBOX based email clients. Additionally, more than one MBOX file can be viewed at a time adding convenience to end user.
External Links
Portable MBOX Viewer
Official Website
Media sanitizing -
Media Sanitizing is the process of removing data, or removing the ability to extract data, from the media on which it is stored. The two main methods used are Clearing or Purging and Destruction. The method chosen will depend largly on whether the media is required to be reused or recycled. However, care should be taken even with some purging methods, degausing for example can destroy the equipment.
Gutmann's 1996 paper on Recovering Overwritten Data suggested that media sanitization (at least by overwriting) was futile as it was always possible to recover the data, although he acknowledged that overwriting multiple times made the recovery difficult, and perhaps prohibitivly expensive. Later he advised that overwriting twice would suffice for most situations - See Epilogue to Gutmann's 1996 paper. The need to overwrite multiple times is regarded as a myth by some who point out that this is only possible with extremly expensive laboratory equipment and that there is no software recovery technique which will recover data overwritten even once (if it is in fact overwritten). See Joukov Papaxenopoulos & Zadok (2006) on "Secure deletion myths, issues, and solutions"
Detailed instructions on media sanitization have been published by NIST as Special Publication 800-88.
Also see
Recovering deleted data
Epilogue to Gutmann's 1996 paper
Sanitizing Tools
External Links
NIST as Special Publication 800-88 Guidelines for Media Sanitization
Joukov Papaxenopoulos & Zadok (2006) Secure Deletion Myths, Issues, and Solutions (subscription is necessary for full text)
Microsoft mail header format -
Microsoft Mail is a Mail User Agent program included with all versions of Windows Vista.
Message-ID: <8A1F11BBD7D7413EAE9E2792741BC6A4@hostname> From: "Username" <username@sendinghost.com> To: "Username" <username@receivinghost.com> Subject: header test Date: Fri, 3 Aug 2007 18:22:27 +0200 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_000B_01C7D5FB.3ED1C170" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Windows Mail 6.0.6000.16386 X-MimeOLE: Produced By Microsoft MimeOLE V6.0.6000.16386