ESOMod:EsoExtractData

EsoExtractData is a Windows command line utility program used to extract data from ESO's MNF/DAT files. It is also available at ESOUI.

Installation
:* Download  or at ESOUI.
:* Unzip into a directory of your choice (preferably a new directory).
:* Run the program from the Windows command line (ex: Start::Run::cmd) or similar shell.

Usage
IMPORTANT NOTE --  Exporting all files from the three ESO MNF files will take several
hours and require over 100GB of free disk space.

:EsoExtractData --help or  EsoExtractData -h
::: View basic program usage and command options:
:EsoExtractData  \path\to\game.mnf \export\path\
::: Extracts the given MNF file and outputs to the given path.
:EsoExtractData  \path\to\game.mnf \export\path\ -z \export\path\zosft.txt -m \export\path\mnf.txt
::: Extracts the MNF file and outputs a ZOSFT and MNF directory listing to the specified files.
:ConvertDDS \export\path\
::: Converts all DDS files recursively in the given path.

Command Line Options
:-h or --help
::: Show basic help.
:-m [filename] or --mnfft [filename]
::: Save the MNF file directory to the given filename.
:-z [filename] or --zosft [filename]
::: Save the ZOSFT file directory to the given filename. 
:-s [fileindex] or --startindex [fileindex]
::: Start exporting from the given file index.
:-e [fileindex] or --endindex [fileindex]
::: Stop exporting at the given file index. 
:-f [fileindex] or --fileindex [fileindex]
::: Only export the file with the given fileindex.
:-a [index] or --archive [index]
::: Only export the given MNF index.
:-b [index] or --beginarchive [index]
::: Start exporting at the given MNF index. 
:-c or --convertdds
::: Convert all DDS files to PNG. Note that this currently crashes on some DDS files.
:-k or --skipsubfiles
::: Don't extract/save files from the MNF/DAT.
:-n  [filename] or --filename [filename]
:::Only extract filenames that match the given string or number. If a number is used it will match the fileindex of the subfile with no extension. 
:::Filenames can include paths however the names are checked both with and without the subfile path. For example, "testfile.DDS" will match both "testfile.dds" 
:::and "path/to/testfile.dds". Match is case insensitive. Wildcards in the filename are not supported. Added in version 0.33.
:-l [filename] or --lang [filename]
::: Convert the given LANG file to "filename.CSV". Also outputs an ID file with the name "filename.id.txt" which is needed when converting a TXT file back to LANG.
:-x [filename] or --createlang [filename]
::: Convert the given CSV file to a LANG file (ex: text.csv to test.lang). The CSV file should be in the same format as one created from the "-l" option including:
:::* One header row: ID,Unknown,Index,Offset,Text
:::* Column order is currently fixed.
:::* Offset column is recomputed when saving.
:::* Note that the values \r, \n and \" in texts are converted to their respective characters.
:::* Text column must be quoted to preserve commas in texts.
:::* Resulting LANG file will be larger than the original due to duplicate texts not being merged.
:-o [filename] or --outputfile [filename]
::: Manually specify the output file for -l and -x commands.
:-p or --pocsv
::: Output a LANG CSV file in a PO (Pootle) compatible format. The CSV will have 3 columns instead of 5:
::::* Location: "ID-UNKNOWN-INDEX"
::::* Source: TEXT
::::* Target: Blank
:-t or --langtext
::: Save the LANG file in a plain text format (one text per line, LFs and CRs will be escaped to \r and \n). If combined with -p and extra blank line will be added between the texts to be compatible with Pootle's txt2po utility.
:-i or --idfile
::: Specifies the ID file to use when converting a TXT file to LANG. An ID file is created when converting a LANG to TXT/CSV and must match the file used when converting the TXT file back to LANG. It is a simple text file with one column containing the unique ID for each text in the file.

:-d [file1] [file2]  or --difflang [file1] [file2]
::: Compares two LANG/TXT/CSV files for differences. Will output files XXX.added.csv, XXX.removed.csv, XXX.changed.csv, and XXX.lang (or XXX.txt and XXX.id.txt if the -t option is used). For column format for the changed CSV file is [id columns...], [new], [old], [translated]. The two input files don't have to be the same type. If using TXT files use the -i1 and -i2 parameters to specify the associated ID files.
        EsoExtractData -d file1.lang file2.lang
        EsoExtractData -d file1.lang file2.lang -p
        EsoExtractData -d file1.lang file2.lang -p -t
        EsoExtractData -d file1.lang file2.csv
        EsoExtractData -d file1.csv file2.lang
        EsoExtractData -d file1.txt file2.lang -i1 file1.id.txt
        EsoExtractData -d file1.lang file2.txt -i2 file2.id.txt
        EsoExtractData -d file1.txt file2.txt -i1 file1.id.txt -i2 file2.id.txt
:::Note: If the ID file used with -i and -i1 is the same you can omit one or the other parameter on the command line. For example, the following commands would be identical:
        EsoExtractData -g tr.txt -i id.txt -d old.txt new.txt -i1 id.txt -i2 new.id.txt
        EsoExtractData -g tr.txt -i id.txt -d old.txt new.txt            -i2 new.id.txt
        EsoExtractData -g tr.txt           -d old.txt new.txt -i1 id.txt -i2 new.id.txt
:-i1 [idfile] or --idfile1 [idfile]
:::Use this to specify the associated ID file if using a TXT file for the first file in a -d parameter. 
:-i2 [idfile] or --idfile2 [idfile]
:::Use this to specify the associated ID file if using a TXT file for the second file in a -d parameter. 
:-g [file] or --origlang [file]
:::Use this with the -d command to specify the source for text for unchanged entries. If a text entry has not been removed or changed the text will come from this file. This can be a LANG, CSV or TXT file. If using a TXT file use the -i parameter to specify the associated ID file.
        EsoExtractData -g orig.lang -d file1.lang file2.lang
        EsoExtractData -g orig.csv -d file1.lang file2.lang
        EsoExtractData -g orig.txt -i orig.id.txt -d file1.lang file2.lang
:--noparsegr2
:::Don't parse any GR2 files for their original filenames. By default all recognized GR2 files are loaded and parsed by the Granny DLL in order to extract and output the file to its original filename.
        EsoExtractData eso.mnf --noparsegr2
:--extractsubfile combined
:::Files that contain compressed record/subfile data are uncompressed and their data output to a single file. The combined file format is output in the following format:

        Header (same 16 bytes as original compressed file)
                dword MagicBytes
                dword Unknown1
                dword NumRecords
                dword Unknown2
        x(0...N) Record Data (variable sized)
                dword MagicBytes = "####"
                dword Index
                dword UncompressedSize1
                dword UncompressedSize2
                dword CompressedSize
                dword Index                	(from index file if found)
                dword OrigFileOffset            (from index file if found)
                dword UncompressedSize
                byte UncompressedData[UncompressedSize]
        
        EsoExtractData eso.mnf --extractsubfile combined
:--extractsubfile seperate
:::Files that contain compressed record/subfile data are uncompressed and their data output into individual files within a new subdirectory. Warning: This creates over 1 million files and adds several hours to the extraction.
        EsoExtractData eso.mnf --extractsubfile seperate
:--luafilelist [filename]
:::If this option is specified a list of all files with a ZOS file table entry is output to the given LUA file in the format:
       [Index] = "Filename",
:::The Index is a number starting at 1 and incrementing for each file. This option also filters filenames based on the optional --fileext parameter.
:--luastartindex [index]
:::Specify the initial file index for the LUA array output by --luafilelist. Default value is 1.
:--fileext [ext]
:::Only extract files matching the given extension. This only applies to files that have a filename specified in the ZOS file table. You don't need to include the period. Technically this just checks if the filename ends with the given string, so TXT would match files File.TXT and FileTXT.

Example
Convert MNF Files
     EsoExtractData c:\input\path\file.mnf c:\output\path\
Convert MNF Files With Directory Files
     EsoExtractData c:\input\path\file.mnf c:\output\path\ -z c:\output\path\zosft.txt -m c:\output\path\mnf.txt
Convert a Specific LANG File to CSV
     EsoExtractData -l file.lang 
Convert a Specific LANG File to CSV With Custom Name 
     EsoExtractData -l file.lang -o newfile.csv
Convert a Specific LANG File to CSV in a PO Compatible Format
     EsoExtractData -l file.lang -p
Convert a Specific LANG File to Text
     EsoExtractData -l file.lang -t
Convert a Specific LANG File to Text in a PO Compatible Format
     EsoExtractData -l file.lang -t -p 
Convert a CSV File to a LANG
     EsoExtractData -x file.csv
Convert a CSV File to a Custom LANG File
     EsoExtractData -x file.csv -o newfile.lang
Convert a PO CSV File to a LANG
     EsoExtractData -x file.csv -p
Convert a Text File to a LANG
     EsoExtractData -x file.txt -i idfile.txt -t
Convert a PO Text File to a LANG
     EsoExtractData -x file.txt -i idfile.txt -t -p
Convert a PO Text File to a Custom LANG File
     EsoExtractData -x file.txt -i idfile.txt -t -p -o newfile.lang

Version History
:* v0.41 -- 17 September 2020
::* Removed the -c/--convertdds option.
::* Added the --luafilelist option.
::* Added the --luastartindex option.
::* Added the -y/--fileext option.

:* v0.40 -- 6 February 2020
::* Changed to a 64 bit build (to support Oodle decompression).
::* Supports the new DAT format introduced in update 25 PTS.
::* Skips empty DAT files which reduces the number of error messages.
::* Added the "--oodleraw" option which saves subfiles in their original Oodle compression format.

:* v0.34 -- 30 May 2019
::* Fix default for the "-n" / "--filename" option.

:* v0.33 -- 19 April 2019
::* Added the "-n" / "--filename" option.

:* v0.32 -- 23 October 2018
::* Fixed crash from null pointer reference received from Granny API.
::* Updated Granny2.dll file which permits more original GR2 filenames to be extracted.
:* v0.31 -- 17 April 2017
::* Fixed extraction of the ZOSFT from ESO MNF in update 14 PTS files. Unsure how this will work with prior file versions however (is fine with update 13 at least).
:* v0.30 -- 18 January 2017
::* Added more matching magic bytes for recognizing GR2 (Granny) files.
::* GR2 model/animation files are output according to their internal original path/filename in  addition to their ZOSFT filename (if it exists) and numeric file index. The original path will be created under the "Granny" path in the base export path.
::* Recognize binary Havok files and assign them with the HKX extension.
::* Recognize file data containing compressed sub-files and assign them the EsoFileData extensions.
::* Recognize file data containing unknown ID data and assign them the EsoIdData extensions.
::* Recognize file data for the PSB2 format (unsure exactly what it is though).
::* Recognize file data containing more text data (including books) and give it the "TextData" extension.
::* Recognize "FFX" file data.
::* Recognize UTF-8 text files starting with the byte order mark EF BB BF and give the extension "TXT".
::* Recognize the unknown XV4 file which seems to be just a DDS with 12 bytes of extra header data and 4 bytes of extra footer. The original file and a DDS version without the extra header is saved.
::* Include the "--noparsegr2" parameter to prevent the parsing of GR2 files to extract their original filename.
::* Added the "--extractsubfile" option for extracting compressed data from some file types:
          --extractsubfile none : Default, does nothing
          --extractsubfile combined : Outputs all compressed files in one large file.
          --extractsubfile seperate : Outputs files individually (Warning: This creates over 1 million files and adds several hours to the extraction).
:* v0.29 -- 5 September 2016
::* Fixed an infinite loop due to a truncated compressed file in file eso0002.dat from the update 12 PTS.
:* v0.28 -- 8 March 2016
::* Fixed a crash when extracting Game.Mnf data from the Thieves Guild DLC release.
:* v0.27 -- 4 February 2016
::* Added support for the 1.9 update on PTS (Thieves Guild DLC) for finding the ZOSFT in the ESO.MNF file. File index for the ZOSFT was changed from 0xFFFFF to 0xFFFFFF.
:* v0.26 -- 19 August 2015
::* When using -d the changed CSV file contains the original translation text supplied with -g if it exists in the last column ([id columns], [new], [old], [translated]).
::* If the ID file used with -i and -i1 is the same you can omit one or the other parameter on the command line. For example, the following commands would be identical:
         EsoExtractData -g tr.txt -i id.txt -d old.txt new.txt -i1 id.txt -i2 new.id.txt
         EsoExtractData -g tr.txt -i id.txt -d old.txt new.txt            -i2 new.id.txt
         EsoExtractData -g tr.txt           -d old.txt new.txt -i1 id.txt -i2 new.id.txt
:* v0.25 -- 28 August 2015
::* Added the "-d" option for comparing LANG/CSV/TXT files.
::* Added the "-i2" option for specifying the second ID file when comparing files.
::* Added the "-g" option for specifying a source text for unchanged entries.
:* v0.24 -- 9 June 2015
::* Added the "-i" option to input an ID text.
::* An ID file (.id.txt) is output when converting a LANG file.
::* Convert a text file along with an ID file to a LANG file.
:* v0.23 -- 11 April 2015
::* Added the -t option for saving LANG files in a plain text format.
:* v0.22 -- 9 April 2015
::* Added the "--posourcetext" to use the source text column (2) in a PO-CSV file when converting it to a LANG file.
::* Assume a PO-CSV file (3 columns) when the -p option is used with -x.
::* Fix the location column (1) when creating a PO-CSV file (offset was used instead of index).
:* v0.21 -- 9 April 2015
::* Escape quotes in text as double-quotes ("") instead of \" in CSV files to import correctly.
::* Added the "-o" option for specifying the output filename for -l/-x commands.
::* Added the "-p" option for outputting LANG CSV files in a PO compatible format.
:* v0.20 -- 9 April 2015
::* All cells are quoted when saving a LANG file as a CSV.
::* Translate DOS linefeeds in LANG file texts as "\r" (was "\n" as previously).
::* Convert a language CSV file back to a LANG file using the "-x" option:
:::* First row must be a header: ID,Unknown,Index,Offset,Text
:::* Column order is currently fixed.
:::* Convert \r, \n and \" to their respective characters.
:::* Text column must be quoted to preserve commas in texts.
:::* Resulting LANG file will be larger than the original due to duplicate texts  not being merged.
:::* Output filename will be the same filename with ".CSV" replaced with ".LANG".
:* v0.18 - 23 November 2014
::* Fixed output of Game.Mnf with filenames.
:* v0.17 - 7 November 2014
::* Name changed from EsoExportMnf to EsoExtractData.
::* Now has better, automatic loading of ZOSFT entries for filename correlation.
::* Fixed incorrect filename assignment to files with the same ID.
::* Added missing , to MNF file table CSV export.
::* Added the "UserData" column to the ZOSFT file table CSV export. Currently is the number of file entries found in the MNF data.
::* Converts any .lang file to a CSV format if it can.
::* Language file now exported in a normal CSV format (with commas and internal double-quotes escaped to \").
::* Added the -b/--beginarchiveoption to start at a specific DAT file index.
::* Added the -l/--lang option to convert a .LANG file to a CSV.
:* v0.16 - March 2014
::* Updated to support the patch for the 14 Mar 2014 beta.
:* v0.15 - February 2014
::* Updated to support the patch for the 8 Feb 2014 beta.

Known Bugs
:* Errors Extracting from ESO0000.DAT -- There are many files from ESO0000.DAT are in an unknown/invalid format and do not extract correctly.
:* Errors Extracting from ESO0204.DAT and Greater -- DAT files with index 204 and above are empty (just the DAT header) even though there are file entries in the MNF file. This results is errors when trying to extract file data from the DAT files that doesn't exist.
:* DDS Conversion With -c -- There are some DDS files that will crash the built-in file loader for an unknown reason. Use the ConvertDDS.bat file included instead.

Converting LANG Files
This is a more in-depth explanation of how to convert LANG files to/from TXT/CSV files related to translating the texts to different languages.

:* Create the Initial Export
         EsoExtractData.exe -l en.lang -o en.orig -p
              Creates: en.orig.csv
         EsoExtractData.exe -l en.lang -o en.orig -t -p
              Creates: en.orig.txt, en.orig.id.txt
::Save these files (LANG, CSV and two TXT) for later use as you will need them when re-importing the translated text. Note that the -p option is for importing the file to [pootle.translatehouse.org Pootle]. You can omit the -p for all commands but make sure to be consistent (a file exported with -p has to be imported with -p).
:* Translate the File
::Import the en.orig.txt or en.orig.csv into your translation program/utility and translate strings as desired. When finished export the translated strings to a new.txt or new.csv file.
:* Create the New LANG File
         EsoExtractData.exe -x new.txt -i en.orig.id.txt -t -p -o new.lang
              or
         EsoExtractData.exe -x new.csv -p -o new.lang
              Creates: new.lang
::The new.lang file can be used in an add-on to "overwrite" the existing LANG file of the game.
:* Update a Partially Translated File
::The game continually updates its text over time so you may wish to update a previously exported TXT/CSV file with all the new/updated text:
         EsoExtractData.exe -g new.txt -i en.orig.id.txt -p -d en.lang en.new.lang -o new.updated
              or
         EsoExtractData.exe -g new.txt -i en.orig.id.txt -p -d en.orig.lang.txt en.new.lang.txt -i1 en.orig.id.txt -i2 en.new.lang.txt -o new.updated
             Creates Files:   new.updated.lang
                              new.updated.lang.txt
                              new.updated.lang.id.txt
                              new.updated.added.csv
                              new.updated.removed.csv
                              new.updated.changed.csv
::The new LANG/TXT/CSV file will have all the translated lines from new.txt that haven't been changed or removed in the new LANG file and have all the new texts added. The changed CSV file will contain the new text followed by the original text in the last column. Be sure to keep the 3 new files new.updated.lang, new.updated.lang.txt, and new.updated.lang.id.txt as you will need them for future translations and updates.             
             

Other
:* Source code is available on GitHub.