Package: libratom
libratomPython library and supporting utilities to parse and process PST and mbox email sources.InstallationLibratom requires Python 3.8 or newer, and can be installed from the Python Package Index. Installing withpipwill automatically install all required dependencies. These dependencies include a version oflibpffthat will be compiled automatically with C tooling during install. A selection of environments we have tested follows:Ubuntu 18.04LTS, 20.04LTS, and 22.04LTS releases require build-essential, python3, python3-pip, and python3-venv packagesRHEL 9 requires python3, python3-pip, python3-devel, and the Development Tools package group.macOS 10.14 (and newer) releases require Xcode 11 (or newer), Xcode CLI tools, and Python 3 installed using Homebrew (or your preferred method)Windows 10/11 releases require Visual Studio Code, Build Tools for Visual Studio, and Python 3 installed using Anaconda 3 (or your preferred method)Need guidance setting up an environment on your platform? Navigate to one of the linked sections below before continuing.Windows environment setupmacOS environment setupUbuntu environment setupRHEL environment setupWestrongly recommendyou create a Python virtual environment prior to installing libratom. If you followed one of the guides above, you should already have one activated. With the environment configured and a Python virtual environment created and activated, run the following commands.Make sure pip is upgraded to the latest version:pipinstall--upgradepipInstall libratom:pipinstalllibratomCLI OverviewLibratom provides a command line interface to run several different tasks. To see available commands, type:(venv)user@host:~$ratom-hFollow one of the section links below for detailed explanations of how the available commands work:Entity extraction: Entity extraction from individual PST and mbox files, or directories containing PST and mbox files. Optionally write message bodies and headers to the database.Model management: Management tool for spaCy language models. Use to display available models and install specific model versions.Scan and report: Quickly scan an email source and generate a report. Optionally write message bodies and headers to the database.Message export: Export selected messages from PST or mbox files as one .eml file per message.Entity extractionTo see detailed help for the entity extraction command, type:(venv)user@host:~$ratomentities-hTo run the extractor with default settings over a PST or mbox file, or a directory containing one or more PST and mbox files, type the following:(venv)user@host:~$ratomentities-p-m/path/to/PST-or-mbox-file-or-directoryNote that the -m flag has been added to this command to write message bodies (stripped of inline attachments and markup) and message headers to the message table. If the -m flag is omitted, these entries will be null. Progress (enabled using the -p flag) is displayed in a bar at the bottom of the window. To terminate a job early and shut down all workers, type Ctrl-C.By default, the tool will install and use the latest spaCy 3 en_core_web_sm model (see themodel managementsection for how to list and install model versions). It will start as many concurrent jobs as there are virtual cores available. Entities are written to a sqlite3 file automatically named using the existing file or directory name and current datetime stamp, and with the following schema:The schema contains five tables. Four tables are used to represent files, messages, attachments, and entities. A fifth table is used to store configuration and environment details relevant to a specific run.In the entity table, text is the entity instance, label_ is the entity type, filepath is the PST or mbox file associated with this entity. Full message and file information for each entity are also available through message_id and file_report_id respectively. Note that pff_identifier (a message ID specific to PST files) will not be populated for messages located in mbox files. Examples of how to query these tables can be found in theInteractive examplessection near the end of this README.Advanced uses of the entity extraction commandThe CLI provides additional flags to tune performance, output location, and verbosity of the tool. Flags that do not take a value may be chained. For example, "-p -v" is equivalent to "-pv" Some example use cases are provided below.The CLI is "quiet" and produces minimal output by default. A single -v flag enables some basic output about job status. To view more detailed output (for example, if you encounter unexpected failures), you can increase the level of output verbosity with -vv (verbosity level 2):(venv)user@host:~$ratomentities-p-m-vv/path/to/PST-or-mbox-file-or-directoryAll remaining examples are presented with verbosity level 1 enabled.To use the latest version of a different entity model, use the --spacy-model flag. The following example directs the tool to use the multi-language model:(venv)user@host:~$ratomentities-p-m-v--spacy-modelxx_ent_wiki_sm/path/to/PST-or-mbox-file-or-directoryThe tool will optimize the number of jobs that may be run concurrently on your system by default, using all available processor cores. To manually set the number of jobs that may be run concurrently, use the -j flag. The following example sets the number of concurrent jobs to 2:(venv)user@host:~$ratomentities-p-m-v-j2/path/to/PST-or-mbox-file-or-directoryTo change the name or location used for the sqlite3 output file, use the -o flag. Specifying a directory will result in the automatically named file being written to that path. Specifying a path that includes a filename will force the use of that filename. In the following example, the sqlite3 database will be named filename.db:(venv)user@host:~$ratomentities-p-m-v-o/path/to/directory/filename.db/path/to/PST-or-mbox-file-or-directoryAs noted earlier, some of these options can be chained for convenience. The following example specifies that progress will be shown, the second level of verbosity will be used, message headers and bodies (stripped of inline attachments and markup) will be written to the database, 4 jobs will be run concurrently and the sqlite3 database will be named filename.db:(venv)user@host:~$ratomentities-pvvm-j4-o/path/to/directory/filename.db/path/to/PST-or-mbox-file-or-directoryModel managementNew spaCy releases are generally accompanied by newly trained models. Using different versions of models over the same collection may produce different results. Depending on your workflow and needs, you may wish to install earlier versions of models, upgrade models that were previously installed, or install multiple models. The model command assists with these tasks.To see detailed help for the model command, type:(venv)user@host:~$ratommodel-hTo see a list of available models, type:(venv)user@host:~$ratommodel-lTo install a specific version of an available model, use the -i and --version flags. For example, to install the 3.4.1 version of en_core_web_sm, type:(venv)user@host:~$ratommodel-ien_core_web_sm--version3.4.1Note that a request to install a specific version will replace any existing version of that model, even if the existing version is newer.Scan and reportTo generate a report (file metadata, message count, and attachment metadata) from an email source (file or directory) in the same sqlite3 databse format as the entity extraction command without actually extracting any entities, use the report command.To see detailed help for the report command, type:(venv)user@host:~$ratomreport-hAs an example, the following command generates a report (showing progress while running) from a single PST or MBOX file, or directory of files:(venv)user@host:~$ratomreport-p/path/to/PST-or-mbox-file-or-directorySimilar to the entities command, adding the -m flag will cause message bodies (stripped of inline attachments and markup) and headers to be written to the message table:(venv)user@host:~$ratomreport-p-m/path/to/PST-or-mbox-file-or-directoryMessage exportThe emldump command generates .eml files from selected messages in one or more PST files. The use case for this command is having to extract messages from multiple PST files fetched from cloud storage onto a local filesystem.
To see its detailed help type:(venv)user@host:~$ratomemldump-hThe required input for emldump is the path to a JSON file listing the source PST files and for each PST file the identifiers of messages to be exported.A sample JSON input is given below:(venv)user@host:~$catfiles.json[{"filename":"andy_zipper_000_1_1.pst","sha256":"7223f79c894f6911ca0fe9a105fa47da0edde31670590d45e2ddfdec37469c4d","id_list":["2203588","2203620","2203652"]},{"filename":"andy_zipper_001_1_1.pst","sha256":"c10d0bb5ac6ee4386c7c990b22f0d337f8f31cfcb685bb1739dd2d8293043134","id_list":["2133380","2133412","2133444"]}]Note that the JSON file contains filenames only. When invoking the command you can specify where those files are via the-lor--locationoption. By default the tool will look for them in the current working directory.You may also specify a base destination folder for the output .eml files via the-oor--outoption. By default the tool will create them under the current working directory, grouped in directories named after eachfilename.For example, the above input file can be used as follows:(venv)user@host:~$ratomemldump-l/tmp/libratom/test_data/RevisedEDRMv1_Complete/andy_zipper/-o/tmp/eml/files.jsonInteractive examplesWe have prepared a selection of Jupyter notebooks to demonstrate core functionality and additional scripts built using libratom. The repository linked below includes instructions for deploying these notebooks in an executable environment using MyBinder:https://github.com/libratom/ratom-notebooksEnvironment Setups for Windows, macOS, and UbuntuWindows environment setupFirst, install Visual Studio Code. Visithttps://code.visualstudio.com/downloadto download and run the 64-bit User Installer for Windows 10. Follow the prompts, accepting all default selections.Download and run the Build Tools for Visual Studio 2022 installer fromhttps://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2022. Follow the prompts until you see a window with a Workloads tab in the top left hand corner.In the Workloads tab, check the box for "C++ build tools". Click the Install button at the bottom right of the window. Once you see "Installation Succeeded!", close the window.Visithttps://www.anaconda.com/products/individualto download and install the 64-bit Python 3.8 or Python 3.9 Anaconda distribution. Find and double-click the downloaded executable and follow the prompts, accepting all default selections.Open the Windows Start Menu, select Anaconda3 (64-bit) and click "Anaconda Prompt (Anaconda3)".In the terminal that appears, create a new virtual environment in which to install libratom. Replace "username" in the prompt with your username. Do not replace the word "name" in the "--name" flag:(base)C:\Users\username>condacreate--nameratomenvType y to confirm any prompts and proceed. Activate the environment:(base)C:\Users\username>condaactivateratomenvType y to confirm and proceed. At the next prompt, install pip:(ratomenv)C:\Users\username>condainstallpipType y to confirm and proceed. At the next prompt, install libratom:(ratomenv)C:\Users\username>pipinstalllibratomLibratom and the ratom CLI tool should now be ready to use.Python virtual environments can be deactivated and reactivated as needed. To deactivate the environment, type:(ratomenv)C:\Users\username>condadeactivateTo remove the environment completely, type:(base)C:\Users\username>condaenvremove-nratomenvNeed additional help getting started with conda? You can find a 20-minute starter guide athttps://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html.macOS environment setupInstall the latest version of Xcode from the App Store. Once Xcode is installed, open a terminal (you can find the terminal app by clicking the Spotlight magnifying glass and typing term).Run the following to install/update the Xcode command line tools:user-macbook:~user$xcode-select--installYou may need to run the following to agree to the Xcode/iOS licence (requires admin privileges):user-macbook:~user$sudoxcodebuild-licenseFollow the instructions at the link below to check your system and install Python 3 if needed:https://installpython3.com/mac/Next, create a new Python 3 virtual environment. Use the instructions in the previous link or create and activate one in your home directory with the following commands:user-macbook:~user$python3-mvenvvenv
user-macbook:~user$sourcevenv/bin/activateFollow the remaining instructions in the Installation section at the top of this README to upgrade pip and install libratom.Ubuntu environment setupTo install and test this software in a new Python virtual environment in Ubuntu 18.04LTS or newer:Make sure Python 3.8 or newer, python3-pip, python3-venv, and build-essential are installed. Open a terminal and type the following command:sudoaptinstallbuild-essentialpython3python3-pippython3-venvCreate and activate a Python virtual environment:python3-mvenvvenvsourcevenv/bin/activateFollow the remaining instructions in the Installation section at the top of this README to upgrade pip and install libratom.RHEL environment setupThese instructions use theDNFpackage manager, and have been tested in RHEL 9. In RHEL 9, you will need to install the python3-pip and python3 devel packages. In a terminal, run the following commands:sudodnfinstallpython3-pip
sudodnfinstallpython3-develReply "y" if prompted during the installations. Next, you'll need to install some development tools, particularly a C compiler. There are several ways to do this, but the simplest is to install the Development Tools group:sudodnfgroupinstall"Development Tools"Create and activate a Python virtual environment:python3-mvenvvenvsourcevenv/bin/activateOnce you have completed these installs, follow the remaining instructions at the top of this README to upgrade pip and install libratom.License(s)Logos, documentation, and other non-software products of the RATOM team are distributed under the terms of Creative Commons 4.0 Attribution. Software items in RATOM repositories are distributed under the terms of the MIT License. See the LICENSE file for additional details.© 2022, The University of North Carolina at Chapel Hill.Development Team and SupportDeveloped by the RATOM team at the University of North Carolina at Chapel Hill.Seehttps://ratom.web.unc.edufor additional project details, staff bios, and news.