Package: txtferret
txtferretIdentify and classify data in your text files with Python.DescriptionDefinition:txtferretA weasel-like mammal that feasts on rodents... and apparently social security numbers,
credit card numbers, or any other data that's in your text or gzipped text files.Use custom regular expressions and sanity checks (ex:luhnalgorithm for account numbers) to find
sensitive data in virtually any size file via your command line.Why use txtferret?  See theHow/why did this come about?section below.Table of ContentsQuick StartConfigurationHow/why did this come about?DevelopmentQuick StartPyPiInstall it$pip3installtxtferretRepoClone it.$gitclonegit@github.com:krayzpipes/txt-ferret.git
$cdtxt-ferretSetup environment.$python3.7-mvenvvenv
$sourcevenv/bin/activateInstall it.(venv)$pythonsetup.pyinstallRun itExample file size:# Decent sized file.$ls-alh|grepmy_test_file.dat
-rw-r--r--1mrferretferrets19GMay711:15my_test_file.datScanning the file.# Scan the file.# Default behavior is to mask the string that was matched.$txtferretscanmy_test_file.dat2019:05:20-22:18:01:-0400Beginningscanfor/home/mrferret/Documents/test_file_1.dat2019:05:20-22:18:18:-0400PASSEDsanityandmatchedregex-/home/mrferret/Documents/test_file_1.dat-Filter:fake_ccn_account_filter,Line712567,String:1001020304050607080942019:05:20-22:19:09:-0400Finishedscanfor/home/mrferret/Documents/test_file_1.dat2019:05:20-22:19:09:-0400SUMMARY:2019:05:20-22:19:09:-0400-Matchedregex,failedsanity:22019:05:20-22:19:09:-0400-Matchedregex,passedsanity:12019:05:20-22:19:09:-0400Finishedin78seconds(~1minutes).Scanning the file with a delimiter.# Break up each line of a CSV into columns by declaring a comma for a delimiter.# Scan each field in the row and return column numbers as well as line numbers.$txtferretscan--delimiter,test_file_1.csv2019:05:20-21:41:57:-0400Beginningscanfor/home/mrferret/Documents/test_file_1.csv2019:05:20-21:44:34:-0400PASSEDsanityandmatchedregex-/home/mrferret/Documents/test_file_1.csv-Filter:fake_ccn_account_filter,Line712567,String:100102030405060708094,Column:1712019:05:20-21:49:16:-0400Finishedscanfor/home/mrferret/Documents/test_file_1.csv2019:05:20-21:49:16:-0400SUMMARY:2019:05:20-21:49:16:-0400-Matchedregex,failedsanity:22019:05:20-21:49:16:-0400-Matchedregex,passedsanity:12019:05:20-21:49:16:-0400Finishedin439seconds(~7minutes).Scan all files in a directory. Write results to a file andstdout# Uses multiprocessing to speed up scans of a bulk group of files$txtferretscan-obulk_testing.log--bulk../test_files/2019:06:09-15:15:27:-0400Detectednon-textfile'/home/mrferret/Documents/test_file_1.dat.gz'...attemptingGZIPmode(slower).2019:06:09-15:15:27:-0400Detectednon-textfile'/home/mrferret/Documents/test_file_2.dat.gz'...attemptingGZIPmode(slower).2019:06:09-15:15:27:-0400Beginningscanfor/home/mrferret/Documents/test_file_1.dat.gz2019:06:09-15:15:27:-0400Beginningscanfor/home/mrferret/Documents/test_file_2.dat.gz2019:06:09-15:15:27:-0400Beginningscanfor/home/mrferret/Documents/test_file_3.dat2019:06:09-15:15:27:-0400PASSEDsanityandmatchedregex-/home/mrferret/Documents/test_file_2.dat.gz-Filter:fake_ccn_account_filter,Line4,String:1001020304050607080942019:06:09-15:15:27:-0400Finishedscanfor/home/mrferret/Documents/test_file_2.dat.gz2019:06:09-15:16:04:-0400PASSEDsanityandmatchedregex-/home/mrferret/Documents/test_file_3.dat-Filter:fake_ccn_account_filter,Line712567,String:1001020304050607080942019:06:09-15:16:51:-0400PASSEDsanityandmatchedregex-/home/mrferret/Documents/test_file_1.dat.gz-Filter:fake_ccn_account_filter,Line712567,String:1001020304050607080942019:06:09-15:17:15:-0400Finishedscanfor/home/mrferret/Documents/test_file_3.dat2019:06:09-15:19:24:-0400Finishedscanfor/home/mrferret/Documents/test_file_1.dat.gz2019:06:09-15:19:24:-0400SUMMARY:2019:06:09-15:19:24:-0400-Scanned3file(s).2019:06:09-15:19:24:-0400-Matchedregex,failedsanity:162019:06:09-15:19:24:-0400-Matchedregex,passedsanity:32019:06:09-15:19:24:-0400-Finishedin236seconds(~3minutes).2019:06:09-15:19:24:-0400FILESUMMARIES:2019:06:09-15:19:24:-0400Matches:1passedsanitychecksand2failed,TimeElapsed:236seconds/~3minutes-/home/mrferret/Documents/test_file_1.dat.gz2019:06:09-15:19:24:-0400Matches:1passedsanitychecksand3failed,TimeElapsed:0seconds/~0minutes-/home/mrferret/Documents/test_file_2.dat.gz2019:06:09-15:19:24:-0400Matches:1passedsanitychecksand2failed,TimeElapsed:107seconds/~1minutes-/home/mrferret/Documents/test_file_3.datConfigurationThere are two ways to configure txt-ferret. You can make changes or add filters through making a
custom configuration file (based on the default YAML file) or you can add some settings via
CLI switches.CLI Switches will always win/take precedence.User-defined configuration file will always beat the default configuration.If any settings are not defined in a user-file configuration by cli switches, then the default
setting will be applied.Txt-ferret comes with a default config which you can dump into any directory
you wish and change it or use it for reference. If you change the file, you have to
specifiy it with the appropriate CLI switch in order for the script to use it. See the
CLI section below.(venv)$txtferretdump-config/file/to/write/to.yamlThere are two sections of the config file:filtersandsettings.FiltersFilters are regular expressions with some metadata. You can use this metadata to
perform sanity checks on regex matches to sift out false positives. (Ex: luhn
algorithm for credit card numbers). You can also mask the output of the matched string
as it is logged to a file or displayed on a terminal.filters:-label:american_express_15_ccnpattern:'((?:34|37)\d{2}(?:(?:[\W_]\d{6}[\W_]\d{5})|\d{11}))'substitute:'[\W_]'exclude_patterns:["dont_match_me","dont_match_me_either"]sanity:luhnmask:index:2,value:XXXXXXXXtype:Credit Card Numberlabel:This will be displayed in the logs when the filter in question has matched a string.pattern:The regular expression which will be used to find data in the file.Regular expression must be compatible with the pythonremodule in the standard library.Be sure that your regular expression only contains ONE and ONLY ONE capture group. For example,
if you are capturing a phone number:Don't do this:'(555-(867|555)-5309)'Do this:'(555-(?:867|555)-5309)'The former example has two capture groups, and inner and an outer.The latter has one capture group (the outer). The inner is a non-capturing
group as defined by starting the capture group with?:.Note: If you run into issues with loading a custom filter, try adding
single-quotes around your regular expression.exclude_patterns:List of regular expressions.If a pattern match also matches any of theexclude_patterns, it will not be included in the results.'substitute:Allows you to define what characters are removed from a string before it is passed to the sanity check(s).Must be a valid regular expression.If missing or empty, the default substitute is[\W_].sanity:This is the algorithm to use with this filter in order to validate the data is really what you're
looking for. For example, 16 digits might just be a random number and not a credit card. Putting the
numbers through theluhnalgorithm will validate they could potentially be an account number and
reduce false positives.You can also pass through a list of strings that represent algorithms as long as the algorithm exists
in the library. If not, please add it and make a pull request!mask:index:This is the position in the matched string in which the masking will begin.valueThe string which will be used to mask the matched string.type:This is basically a description of the 'type' of data you're looking for with this filter.Settingssettings:mask:Nolog_level:INFOsummarize:Nooutput_file:show_matches:Yesdelimiter:ignore_columns:[1,5,6]file_encoding:'utf-8'bulkThis setting is accessible via CLI arguments-bor--bulk.When those switches are used, pass the directory to scan instead of a single file name.Example:$txtferretscan--bulk/home/mrferret/DocumentsmaskIf set to true, the mask value defined in the filter will be used to mask the data during output.If no mask is set for a filter, the program will mask with a default mask value.This is set to 'true' by default.CLI- The-mswitch can be used to turn on masking'$txtferretscan-m../fake_ccn_data.txt2017:05:20-00:24:52:-0400PASSEDsanityandmatchedregex-Filter:fake_ccn_account_filter,Line1,String:10XXXXXXXXXXXXXXXXXXX

$txtferretscan../fake_ccn_data.txt2017:05:20-00:26:18:-0400PASSEDsanityandmatchedregex-Filter:fake_ccn_account_filter,Line1,String:100102030405060708094summarizeIf set to true, the script will only output a list of three data points:The number of matches that did not pass sanity checks.The number of matches that did pass sanity checks.The time it took to finish searching the file.CLI- The-sswitch will kickoff the summary.$txtferretscan../fake_ccn_data.txt2019:05:20-00:36:00:-0400PASSEDsanityandmatchedregex-Filter:fake_ccn_account_filter,Line1,String:100102030405060708094$txtferretscan-s../fake_ccn_data.txt2019:05:20-01:05:29:-0400SUMMARY:2019:05:20-01:05:29:-0400-Matchedregex,failedsanity:12019:05:20-01:05:29:-0400-Matchedregex,passedsanity:12019:05:20-01:05:29:-0400Finishedin0seconds(~0minutes)output_fileAdd the absolute path to a file for script output.Note- File results are put into individual log files per file scanned.CLI- Use the-oswitch to set an output file.$txtferretscan-omy_output.logfile_to_scan.txtshow_matchesIf this is set to 'No', then it will redact the matched string all-together in output.
Be careful that you do not override this setting via a CLI switch unless it is on purpose.delimiterDefine a delimiter. This delimiter will be used to split each line from the txt file into
columns. The output of the script will provide you with the column in which the regex matched in
addition to the line number.CAUTION:Searching by columns GREATLY slows down the script since you are applying a regular
expression to each column instead of the entire line.You can define a byte-code delimiter by usingbfollowed by the code. For example,b1will
use Start of Header as a delimiter (\x01 in hex)CLI- Use the-dswitch to set a delimiter and scan per column instead of line.$txtferretscan../fake_ccn_data.txt2019:05:20-00:36:00:-0400PASSEDsanityandmatchedregex-Filter:fake_ccn_account_filter,Line1,String:100102030405060708094# Comma delimiter$txtferretscan-d,../fake_ccn_CSV_file.csv2019:05:20-01:12:18:-0400PASSEDsanityandmatchedregex-Filter:fake_ccn_account_filter,Line1,String:100102030405060708094,Column:3ignore_columnsThis setting is ignored if thedelimitersetting or switch is not set.Add a list of integers and txtferret will skip those columns.Ifignore_columns: [2, 6]is configured and a csv row ishello,world,how,are,you,doing,today, thenworldanddoingwill not be scanned but will be ignored.This is particularly useful in columnar datasets when you know there is a column that is full of false positives.file_encodingTwo uses:Used to encode yourdelimitervalue to the appropriate encoding of your file.Used to encode the data matched in the file before being applied to sanity check.Default value is'utf-8'How/why did this come about?There are a few shortcomings with commercial Data Loss Prevention (DLP) products:They often rely on context. It would be too noisy for a commercial solution to alert every time it
matched a sixteen digit string as not all of their customers handle credit card data.Many vendors don't have a cheap method of handling large files over 1 GB. Even less can handle
files that are many GBs in size.Some DLP solutions will only classify data files based on the first so many megabytes.Many DLP solutions are black boxes of magic that do not give you say in what they are looking for
in your data or how they know what they're looking at.Txtferret was born out after realizing some of these limitations. It isn't perfect, but it's a great
sanity check which can be paired with a DLP solution. Here are some things it was designed to do:Virtually no size limitationCan run against any size file as long as you can fit it on a drive.It's python... so... no speed guarantee on huge files, but at least it will eventually get it done.
We've found that txtferret can scan a ~20 GB file between 1 and 3 minutes on our systems.CustomizableDefine your own regular expressions and pair them with a sanity check. For example, using the
built-inluhnalgorithm will sift out many false positives for credit card numbers. The matched
credit card number will be run through theluhnalgorithm first. If it doesn't pass, it is discarded.No context neededYeah, this can cause a lot of false positives with certain files; However,
if you're dealing with a file that doesn't contain context like 'VISA' or 'CVE', then you need
to start somewhere.Helpful outputIndicates which line the string was found.Indicates which column if you've defined a delimiter (ex: comma for CSV files).You can choose to mask your output data to make sure you're not putting sensitive data
into your log files or outputting them to your terminal.It's freeNo contractsNo outrageous licensing per GB of data scanned.You can contribute!ReleasesVersion 0.3.2 - 2019-09-05Fixed cli setting order and dependency bugs.Version 0.3.1 --> I goofed up the uploadVersion 0.3.0a - 2019-09-05Removed log level switch. Only matches are shown now.Addedexclude_patternsto filtersAdjusted result format to tab-delimited output.Bulk scans now write results to separate files instead of all in one file.Version 0.2.1 - 2019-08-11Fixed allowed keys bug introduced in v0.2.0Version 0.2.0 - 2019-08-05Changedtokenizetomaskbecause tokenize was a lie.. it's masking.Replaced--no-tokenizeswitch with--maskswitch.Turned masking off by default.Version 0.1.3 - 2019-08-05Addedfile_encodingsetting for multi-encoding support.Reads in bytes and assumes'utf-8'encoding by default.Version 0.1.2 - 2019-08-01Fixed bug with regex when reading gzipped files.Version 0.1.1 - 2019-07-30Addedsubstituteoption to filters.Version 0.1.0 - 2019-07-30Removed theconfig-overrideoption.Addedignore_columnssetting.Version 0.0.4 - 2019-06-09Added bulk file scanning by the--bulkswitch.Added multiprocessing for bulk scanning.Version 0.0.3 - 2019-06-01Added gzip detection and support.DevelopmentSome info about development.Running Tests$pytesttxt-ferret/tests/ContributingProcessCreate an issue.Fork the repo.Do your work.WRITE TESTSMake a pull request.Preferably, include the issue # in the pull request.StyleBlack for formatting.Pylint for linting.LicenseSeeLicense