Repository: krayzpipes/txt-ferret

Issue:
Add bulk file scan.
Add a switch and/or config option that would allow all files in a directory to be scanned and then an aggregate report be generated.

Discussion:

Patch:
diff --git a/README.md b/README.md
index 4c7f434..5745621 100644
--- a/README.md
+++ b/README.md
@@ -26,6 +26,7 @@ Why use txtferret?  See the __How/why did this come about?__ section below.
     ```bash
     $ pip3 install txtferret
     ```
+
 ### Repo
 1. Clone it.
     ```bash
@@ -41,6 +42,7 @@ Why use txtferret?  See the __How/why did this come about?__ section below.
     ```bash
     (venv) $ python setup.py install
     ```
+
 ### Run it
 - Example file size:
     ```bash
@@ -49,13 +51,16 @@ Why use txtferret?  See the __How/why did this come about?__ section below.
     $ ls -alh | grep my_test_file.dat
     -rw-r--r--  1 mrferret ferrets  19G May  7 11:15 my_test_file.dat
     ```
+
 - Scanning the file.
     ```bash
     # Scan the file.
     # Default behavior is to mask the string that was matched.
     
     $ txtferret scan my_test_file.dat
-    2019:05:20-22:18:18:-0400 PASSED sanity and matched regex - Filter: visa_16_ccn, Line 712567, String: 4XXXXXXXXXXXXXXX
+    2019:05:20-22:18:01:-0400 Beginning scan for /home/mrferret/Documents/test_file_1.dat
+    2019:05:20-22:18:18:-0400 PASSED sanity and matched regex - /home/mrferret/Documents/test_file_1.dat - Filter: visa_16_ccn, Line 712567, String: 4XXXXXXXXXXXXXXX
+    2019:05:20-22:19:09:-0400 Finished scan for /home/mrferret/Documents/test_file_1.dat
     2019:05:20-22:19:09:-0400 SUMMARY:
     2019:05:20-22:19:09:-0400   - Matched regex, failed sanity: 2
     2019:05:20-22:19:09:-0400   - Matched regex, passed sanity: 1
@@ -66,13 +71,42 @@ Why use txtferret?  See the __How/why did this come about?__ section below.
     # Break up each line of a CSV into columns by declaring a comma for a delimiter.
     # Scan each field in the row and return column numbers as well as line numbers.
     
-    $ txtferret scan --delimiter , my_test_file.dat
-    2019:05:20-21:44:34:-0400 PASSED sanity and matched regex - Filter: visa_16_ccn, Line 712567, String: 4XXXXXXXXXXXXXXX, Column: 171
+    $ txtferret scan --delimiter , test_file_1.csv
+    2019:05:20-21:41:57:-0400 Beginning scan for /home/mrferret/Documents/test_file_1.csv
+    2019:05:20-21:44:34:-0400 PASSED sanity and matched regex - /home/mrferret/Documents/test_file_1.csv - Filter: visa_16_ccn, Line 712567, String: 4XXXXXXXXXXXXXXX, Column: 171
+    2019:05:20-21:49:16:-0400 Finished scan for /home/mrferret/Documents/test_file_1.csv
     2019:05:20-21:49:16:-0400 SUMMARY:
     2019:05:20-21:49:16:-0400   - Matched regex, failed sanity: 2
     2019:05:20-21:49:16:-0400   - Matched regex, passed sanity: 1
     2019:05:20-21:49:16:-0400 Finished in 439 seconds (~7 minutes).
     ```
+    - Scan all files in a directory. Write results to a file and `stdout`
+    ```bash
+    # Uses multiprocessing to speed up scans of a bulk group of files
+ 
+    $ txtferret scan -o bulk_testing.log --bulk ../test_files/
+    2019:06:09-15:15:27:-0400 Detected non-text file '/home/mrferret/Documents/test_file_1.dat.gz'... attempting GZIP mode (slower).
+    2019:06:09-15:15:27:-0400 Detected non-text file '/home/mrferret/Documents/test_file_2.dat.gz'... attempting GZIP mode (slower).
+    2019:06:09-15:15:27:-0400 Beginning scan for /home/mrferret/Documents/test_file_1.dat.gz
+    2019:06:09-15:15:27:-0400 Beginning scan for /home/mrferret/Documents/test_file_2.dat.gz
+    2019:06:09-15:15:27:-0400 Beginning scan for /home/mrferret/Documents/test_file_3.dat
+    2019:06:09-15:15:27:-0400 PASSED sanity and matched regex - /home/mrferret/Documents/test_file_2.dat.gz - Filter: visa_16_ccn, Line 4, String: 4XXXXXXXXXXXXXXX026
+    2019:06:09-15:15:27:-0400 Finished scan for /home/mrferret/Documents/test_file_2.dat.gz
+    2019:06:09-15:16:04:-0400 PASSED sanity and matched regex - /home/mrferret/Documents/test_file_3.dat - Filter: visa_16_ccn, Line 712567, String: 4XXXXXXXXXXXXXXX
+    2019:06:09-15:16:51:-0400 PASSED sanity and matched regex - /home/mrferret/Documents/test_file_1.dat.gz - Filter: visa_16_ccn, Line 712567, String: 4XXXXXXXXXXXXXXX
+    2019:06:09-15:17:15:-0400 Finished scan for /home/mrferret/Documents/test_file_3.dat
+    2019:06:09-15:19:24:-0400 Finished scan for /home/mrferret/Documents/test_file_1.dat.gz
+    2019:06:09-15:19:24:-0400 SUMMARY:
+    2019:06:09-15:19:24:-0400   - Scanned 3 file(s).
+    2019:06:09-15:19:24:-0400   - Matched regex, failed sanity: 16
+    2019:06:09-15:19:24:-0400   - Matched regex, passed sanity: 3
+    2019:06:09-15:19:24:-0400   - Finished in 236 seconds (~3 minutes).
+    2019:06:09-15:19:24:-0400 FILE SUMMARIES:
+    2019:06:09-15:19:24:-0400 Matches: 1 passed sanity checks and 2 failed, Time Elapsed: 236 seconds / ~3 minutes - /home/mrferret/Documents/test_file_1.dat.gz
+    2019:06:09-15:19:24:-0400 Matches: 1 passed sanity checks and 3 failed, Time Elapsed: 0 seconds / ~0 minutes - /home/mrferret/Documents/test_file_2.dat.gz
+    2019:06:09-15:19:24:-0400 Matches: 1 passed sanity checks and 2 failed, Time Elapsed: 107 seconds / ~1 minutes - /home/mrferret/Documents/test_file_3.dat
+    ```
+    
 
 # Configuration
 
@@ -154,7 +188,13 @@ settings:
   show_matches: Yes
   delimiter:
 ```
-
+- **bulk**
+    - This setting is accessible via CLI arguments `-b` or `--bulk`.
+    - When those switches are used, pass the directory to scan instead of a single file name.
+    - Example:
+    ```bash
+    $ txtferret scan --bulk /home/mrferret/Documents
+    ```
 - **tokenize**
     - If set to true, the token mask defined in the filter will be used to mask the data during output.
     - If no mask is set for a filter, the program will tokenize with a default mask.
diff --git a/requirements.txt b/requirements.txt
index c9ec897..2b7b9fb 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,11 +1,44 @@
 ansimarkup==1.4.0
 appdirs==1.4.3
+astroid==2.2.5
+atomicwrites==1.3.0
 attrs==19.1.0
 better-exceptions-fork==0.2.1.post6
 black==19.3b0
+bleach==3.1.0
+boto3==1.9.162
+botocore==1.12.162
+certifi==2019.3.9
+chardet==3.0.4
 Click==7.0
 colorama==0.4.1
+docutils==0.14
+idna==2.8
+isort==4.3.20
+jmespath==0.9.4
+lazy-object-proxy==1.4.1
 loguru==0.2.5
+mccabe==0.6.1
+more-itertools==7.0.0
+pkg-resources==0.0.0
+pkginfo==1.5.0.1
+pluggy==0.11.0
+py==1.8.0
 Pygments==2.3.1
+pylint==2.3.1
+pytest==4.5.0
+python-dateutil==2.8.0
 PyYAML==5.1
+readme-renderer==24.0
+requests==2.22.0
+requests-toolbelt==0.9.1
+s3transfer==0.2.1
+six==1.12.0
 toml==0.10.0
+tqdm==4.32.1
+twine==1.13.0
+typed-ast==1.4.0
+urllib3==1.25.3
+wcwidth==0.1.7
+webencodings==0.5.1
+wrapt==1.11.1
diff --git a/setup.py b/setup.py
index 7c986b6..c16247c 100644
--- a/setup.py
+++ b/setup.py
@@ -14,7 +14,7 @@ from setuptools import setup, find_packages
 from codecs import open
 from os import path
 
-__version__ = "0.0.3"
+__version__ = "0.0.4"
 description = "Scan text files for sensitive (or non-sensitive) data."
 
 here = path.abspath(path.dirname(__file__))
diff --git a/src/txtferret/cli.py b/src/txtferret/cli.py
index 67366fd..59b76c8 100644
--- a/src/txtferret/cli.py
+++ b/src/txtferret/cli.py
@@ -1,3 +1,9 @@
+"""Handle CLI tool configuration and commands."""
+
+import copy
+from datetime import datetime
+import multiprocessing as mp
+import pathlib
 import sys
 
 import click
@@ -7,12 +13,117 @@ from ._config import load_config, save_config
 from .core import TxtFerret
 
 
-def _get_settings_dict(*args):
-    pass
+def set_logger(**cli_kwargs):
+    """Configured logger.
+
+    Sets up handlers for stdout and for output file if the user
+    passed an output file name.
+
+    Sets up message format and level.
+
+    :param cli_kwargs: The key/value pairs depicting the CLI arguments
+        given by the user.
+    """
+    # Setup basic log config including a sink for stdout.
+    log_config = {
+        "handlers": [
+            {
+                "sink": sys.stdout,
+                "format": "<lvl>{time:YYYY:MM:DD-HH:mm:ss:ZZ} {message}</lvl>",
+                "level": cli_kwargs["log_level"],
+                "enqueue": True,
+            }
+        ]
+    }
+
+    output_file = cli_kwargs["output_file"]
+
+    # If output file is defined, create a sink for it and add it to
+    # the logger.
+    if output_file is not None:
+        output_sink = {
+            "sink": output_file,
+            "serialize": False,
+            "format": "{time:YYYY:MM:DD-HH:mm:ss:ZZ} {message}",
+            "level": cli_kwargs["log_level"],
+            "enqueue": True,
+        }
+        log_config["handlers"].append(output_sink)
+
+    logger.configure(**log_config)
+
+
+def prep_config(loader=None, **cli_kwargs):
+    """Return a final config file to be sent to TxtFerret."""
+    _loader = loader or load_config
+    file_name = cli_kwargs["config_file"]
+    override = cli_kwargs["config_override"]
+    config = _loader(yaml_file=file_name, default_override=override)
+    config["cli_kwargs"] = {**cli_kwargs}
+    return config
+
+
+def bootstrap(config, test_class=None):
+    """Bootstrap scanning a single file and return summary."""
+    ferret_class = test_class or TxtFerret
+    ferret = ferret_class(config)
+    ferret.scan_file()
+    return ferret.summary()
+
+
+def get_files_from_dir(directory=None):
+    """Return list of absolute file names."""
+    path = pathlib.Path(directory)
+    file_names = [str(item.resolve()) for item in path.iterdir() if item.is_file()]
+    return file_names
+
+
+def get_totals(results=None):
+    """Return counts for failures and successes."""
+    _total_failures = 0
+    _total_passes = 0
+
+    for result in results:
+        _total_failures += result.get("failures")
+        _total_passes += result.get("passes")
+
+    return _total_failures, _total_passes
+
+
+def log_summary(result=None, file_count=None, results=None):
+    """Log summary to logger."""
+    failures = result.get("failures")
+    passes = result.get("passes")
+    logger.info("SUMMARY:")
+    logger.info(f"  - Scanned {file_count} file(s).")
+    logger.info(f"  - Matched regex, failed sanity: {failures}")
+    logger.info(f"  - Matched regex, passed sanity: {passes}")
+
+    seconds = result.get("time")
+    minutes = seconds // 60
+
+    logger.info(f"  - Finished in {seconds} seconds (~{minutes} minutes).")
+
+    if results is None:
+        return
+
+    logger.info("FILE SUMMARIES:")
+    for _result in results:
+        _name = _result.get("file_name")
+        _failures = _result.get("failures")
+        _passes = _result.get("passes")
+        _seconds = _result.get("time")
+        _mins = _seconds // 60
+
+        logger.info(
+            f"Matches: {_passes} passed sanity checks and {_failures} failed, "
+            f"Time Elapsed: {_seconds} seconds / ~{_mins} minutes - {_name}"
+        )
 
 
 @click.group()
 def cli():
+    """Placeholder"""
     pass
 
 
@@ -51,59 +162,64 @@ def cli():
     default="",
     help="Delimiter to use for field parsing instead of line parsing.",
 )
+@click.option("--bulk", "-b", is_flag=True, help="Scan multiple files in a directory.")
 @click.argument("file_name")
 def scan(**cli_kwargs):
     """Kicks off scanning of user-defined file(s)."""
+
+    config = prep_config(**cli_kwargs)
+
     set_logger(**cli_kwargs)
-    ferret = TxtFerret(**cli_kwargs)
-    ferret.scan_file()
 
+    if not cli_kwargs["bulk"]:
 
-@click.command()
-@click.argument("file_name")
-def dump_config(file_name):
-    """Writes default config to user-specified file location."""
-    config = load_config()
-    save_config(config, file_name)
+        result = bootstrap(config)
 
+        log_summary(result=result, file_count=1)
 
-cli.add_command(scan)
-cli.add_command(dump_config)
+    else:
 
+        start = datetime.now()
 
-def set_logger(**cli_kwargs):
-    """Configured logger.
+        file_names = get_files_from_dir(directory=cli_kwargs["file_name"])
+        configs = []
 
-    Sets up handlers for stdout and for output file if the user
-    passed an output file name.
+        # Generate a config for each file name which can be passed to
+        # multiprocessing...
+        for file_ in file_names:
+            temp_config = copy.deepcopy(config)
+            temp_config["cli_kwargs"]["file_name"] = file_
+            configs.append(temp_config)
 
-    Sets up message format and level.
+        # Devy out the work to available CPUs
+        cpus = mp.cpu_count()
+        with mp.Pool(cpus) as p:
+            results = p.map(bootstrap, configs)
 
-    :param cli_kwargs: The key/value pairs depicting the CLI arguments
-        given by the user.
-    """
-    # Setup basic log config including a sink for stdout.
-    log_config = {
-        "handlers": [
-            {
-                "sink": sys.stdout,
-                "format": "<lvl>{time:YYYY:MM:DD-HH:mm:ss:ZZ} {message}</lvl>",
-                "level": cli_kwargs["log_level"],
-            }
-        ]
-    }
+        end = datetime.now()
 
-    output_file = cli_kwargs["output_file"]
+        total_failures, total_passes = get_totals(results)
 
-    # If output file is defined, create a sink for it and add it to
-    # the logger.
-    if output_file is not None:
-        output_sink = {
-            "sink": output_file,
-            "serialize": False,
-            "format": "{time:YYYY:MM:DD-HH:mm:ss:ZZ} {message}",
-            "level": cli_kwargs["log_level"],
+        total_scanned = len(results)
+
+        delta = end - start
+
+        total_result = {
+            "failures": total_failures,
+            "passes": total_passes,
+            "time": delta.seconds,
         }
-        log_config["handlers"].append(output_sink)
 
-    logger.configure(**log_config)
+        log_summary(result=total_result, file_count=total_scanned, results=results)
+
+
+@click.command()
+@click.argument("file_name")
+def dump_config(file_name):
+    """Writes default config to user-specified file location."""
+    config = load_config()
+    save_config(config, file_name)
+
+
+cli.add_command(scan)
+cli.add_command(dump_config)
diff --git a/src/txtferret/core.py b/src/txtferret/core.py
index 102e68a..8163b91 100644
--- a/src/txtferret/core.py
+++ b/src/txtferret/core.py
@@ -7,11 +7,13 @@ import re
 
 from loguru import logger
 
-from ._config import load_config, _allowed_settings_keys
+from ._config import _allowed_settings_keys
 from ._sanity import sanity_check
 
 
-def tokenize(clear_text, mask, index, tokenize=True, show_matches=False):
+def tokenize(
+        clear_text, mask, index, tokenize=True, show_matches=False, tokenize_func=None,
+):
     """Return string as redacted, tokenized format, or clear text.
 
     :param clear_text: The text to be tokenized.
@@ -36,7 +38,9 @@ def tokenize(clear_text, mask, index, tokenize=True, show_matches=False):
     if not tokenize:
         return clear_text
 
-    return _get_tokenized_string(clear_text, mask, index)
+    tokenize_function = tokenize_func or _get_tokenized_string
+
+    return tokenize_function(clear_text, mask, index)
 
 
 def _get_tokenized_string(text, mask, index):
@@ -182,17 +186,16 @@ class TxtFerret:
     :attribute filters: List of filters to be used during the file scan.
     """
 
-    def __init__(self, file_name=None, config_file=None, config_=None, **cli_settings):
+    def __init__(self, config):
         """Initialize the TxtFerret object."""
-        config = config_ or load_config(
-            yaml_file=config_file, default_override=cli_settings["config_override"]
-        )
-        self.file_name = file_name
+        cli_settings = config["cli_kwargs"]
+
+        self.file_name = cli_settings["file_name"]
         self.gzip = gzipped_file_check(self.file_name)
 
         if self.gzip:
             logger.info(
-                f"Detected non-text file '{file_name}'... "
+                f"Detected non-text file '{self.file_name}'... "
                 f"attempting GZIP mode (slower)."
             )
 
@@ -206,6 +209,8 @@ class TxtFerret:
         self.failed_sanity = 0
         self.passed_sanity = 0
 
+        self._time_delta = None
+
         self.filters = [
             Filter(filter_dict=filter_, gzip=self.gzip) for filter_ in config["filters"]
         ]
@@ -245,6 +250,14 @@ class TxtFerret:
 
             setattr(self, setting, value)
 
+    def summary(self):
+        return {
+            "file_name": self.file_name,
+            "failures": self.failed_sanity,
+            "passes": self.passed_sanity,
+            "time": self._time_delta.seconds,
+        }
+
     def _get_file_size(self):
         """Return file size in Megabytes."""
         file_ = Path(self.file_name)
@@ -256,10 +269,13 @@ class TxtFerret:
 
         :param file_name: Name of the file to scan.
         """
+
         start = datetime.now()
 
         file_to_scan = file_name or self.file_name
 
+        logger.info(f"Beginning scan for {file_to_scan}")
+
         if not self.gzip:
             _open = open
         else:
@@ -280,16 +296,9 @@ class TxtFerret:
                 self._scan_non_delimited_line(line, index)
 
         end = datetime.now()
-        delta = end - start
-
-        logger.info("SUMMARY:")
-        logger.info(f"  - Matched regex, failed sanity: {self.failed_sanity}")
-        logger.info(f"  - Matched regex, passed sanity: {self.passed_sanity}")
-
-        seconds = delta.seconds
-        minutes = delta.seconds // 60
+        self._time_delta = end - start
 
-        logger.info(f"Finished in {seconds} seconds (~{minutes} minutes).")
+        logger.info(f"Finished scan for {self.file_name}")
 
     def _scan_delimited_line(self, line, index):
         """Scan a delimited line.
@@ -304,7 +313,10 @@ class TxtFerret:
             # For example... Start Of Header (SOH).
             delimiter = _byte_code_to_string(self.delimiter)
 
-            columns = line.split(delimiter)
+            if not self.gzip:
+                columns = line.split(delimiter)
+            else:
+                columns = line.split(delimiter.encode('utf-8'))
 
             # Look for matches in each column.
             for i, column in enumerate(columns):
@@ -324,11 +336,11 @@ class TxtFerret:
             for column_number, column_match_list in column_map.items():
                 for column_match in column_match_list:
 
-                    if not test_sanity(filter_, column_match):
+                    if not sanity_test(filter_, column_match):
                         self.failed_sanity += 1
 
                         if not self.summarize:
-                            log_failure(filter_, index)
+                            log_failure(self.file_name, filter_, index)
 
                         continue
 
@@ -344,7 +356,11 @@ class TxtFerret:
 
                     if not self.summarize:
                         log_success(
-                            filter_, index, string_to_log, column=int(column_number)
+                            self.file_name,
+                            filter_,
+                            index,
+                            string_to_log,
+                            column=int(column_number),
                         )
 
     def _scan_non_delimited_line(self, line=None, index=None):
@@ -360,10 +376,10 @@ class TxtFerret:
                 continue
 
             for match in matches:
-                if not test_sanity(filter_, match):
+                if not sanity_test(filter_, match):
                     self.failed_sanity += 1
                     if not self.summarize:
-                        log_failure(filter_, index)
+                        log_failure(self.file_name, filter_, index)
                     continue
 
                 self.passed_sanity += 1
@@ -377,25 +393,28 @@ class TxtFerret:
                 )
 
                 if not self.summarize:
-                    log_success(filter_, index, string_to_log)
+                    log_success(self.file_name, filter_, index, string_to_log)
 
 
-def test_sanity(filter_, text):
+def sanity_test(filter_, text, sanity_func=None):
     """Return bool depending on if text passes the sanity check.
 
     :param filter_: Filter object.
     :param text: The text being tested by the sanity check.
+    :sanity_func: Used for tests.
 
     :return: True or False - Depending on if sanity check passed
         or not.
     """
+    _sanity_checker = sanity_func or sanity_check
+
     for algorithm_name in filter_.sanity:
-        if not sanity_check(algorithm_name, text):
+        if not _sanity_checker(algorithm_name, text):
             return False
     return True
 
 
-def log_success(filter_, index, string_, column=None):
+def log_success(file_name, filter_, index, string_, column=None):
     """Log success messages.
 
     :param filter_: The Filter object.
@@ -404,20 +423,20 @@ def log_success(filter_, index, string_, column=None):
     :param column: Column which the filter matched some text.
     """
     matched_string = string_
-    if column:
+    if column is not None:
         message = (
-            f"PASSED sanity and matched regex - Filter: {filter_.label}, "
+            f"PASSED sanity and matched regex - {file_name} - Filter: {filter_.label}, "
             f"Line {index + 1}, String: {matched_string}, Column: {column + 1}"
         )
     else:
         message = (
-            f"PASSED sanity and matched regex - Filter: {filter_.label}, "
+            f"PASSED sanity and matched regex - {file_name} - Filter: {filter_.label}, "
             f"Line {index + 1}, String: {matched_string}"
         )
     logger.info(message)
 
 
-def log_failure(filter_, index, column=None):
+def log_failure(file_name, filter_, index, column=None):
     """Log success messages.
 
     :param filter_: The Filter object.
@@ -426,12 +445,12 @@ def log_failure(filter_, index, column=None):
     """
     if column:
         message = (
-            f"FAILED sanity and matched regex - Filter: {filter_.label}, "
+            f"FAILED sanity and matched regex - {file_name} - Filter: {filter_.label}, "
             f"Line: {index + 1}, Column: {column + 1}"
         )
     else:
         message = (
-            f"FAILED sanity and matched regex - Filter: {filter_.label}, "
+            f"FAILED sanity and matched regex - {file_name} - Filter: {filter_.label}, "
             f"Line: {index + 1}"
         )
     logger.debug(message)


Test patch:
diff --git a/tests/cli/test_cli.py b/tests/cli/test_cli.py
new file mode 100644
index 0000000..bd0137e
--- /dev/null
+++ b/tests/cli/test_cli.py
@@ -0,0 +1,60 @@
+
+from txtferret.cli import prep_config, bootstrap, get_totals
+
+
+def test_prep_config():
+
+    def stub_loader(yaml_file=None, default_override=False):
+        return {
+            "yaml_file": yaml_file,
+            "config_override": default_override,
+        }
+
+    fake_cli_kwargs = {
+        "config_file": "my_test_file",
+        "config_override": True,
+    }
+
+    final_config = {
+        "cli_kwargs": {**fake_cli_kwargs},
+        "yaml_file": "my_test_file",
+        "config_override": True,
+    }
+
+    assert prep_config(loader=stub_loader, **fake_cli_kwargs) == final_config
+
+
+def test_bootstrap():
+
+    class StubClass:
+        def __init__(self, _config):
+            pass
+        def scan_file(self):
+            pass
+        def summary(self):
+            return {
+                "file_name": "hello.txt",
+                "failures": 5,
+                "passes": 5,
+                "time": 28,
+            }
+
+    expected = {
+        "file_name": "hello.txt",
+        "failures": 5,
+        "passes": 5,
+        "time": 28,
+    }
+
+    config = None
+
+    assert bootstrap(config, StubClass) == expected
+
+
+def test_get_totals():
+    results = [
+        {"failures": 2, "passes": 5},
+        {"failures": 3, "passes": 10},
+    ]
+
+    assert get_totals(results) == (5, 15)
diff --git a/tests/core/test_core.py b/tests/core/test_core.py
index 8679f1a..ce47df1 100644
--- a/tests/core/test_core.py
+++ b/tests/core/test_core.py
@@ -1,6 +1,14 @@
 from contextlib import contextmanager
 
-from txtferret.core import gzipped_file_check
+import pytest
+
+from txtferret.core import (
+    gzipped_file_check,
+    tokenize,
+    _get_tokenized_string,
+    _byte_code_to_string,
+    sanity_test,
+)
 
 
 def test_gzipped_file_check_return_true():
@@ -33,3 +41,84 @@ def test_gzipped_file_check_return_false():
         yield FileHandlerStub()
 
     assert gzipped_file_check("f.txt", _opener=opener_stub_no_error) == False
+
+
+def test_tokenize_not_show_matches():
+
+    assert tokenize("hello", "XXX", 0, show_matches=False) == "REDACTED"
+
+
+def test_tokenize_return_clear_text():
+
+    assert tokenize("hello", "XXX", 0, tokenize=False, show_matches=True) == "hello"
+
+
+def test_tokenize_runs_tokenization_function():
+
+    def stub_func(arg1, arg2, arg3):
+        return "stub was called"
+
+    assert tokenize("hello", "XXX", 0, show_matches=True, tokenize_func=stub_func)\
+           == "stub was called"
+
+
+def test_tokenize_for_byte_return():
+    def stub_func(arg1, arg2, arg3):
+        return (arg1, arg2, arg3)
+
+    assert tokenize(b"hello", b"XXX", 0, show_matches=True, tokenize_func=stub_func)\
+           == ("hello", "XXX", 0)
+
+
+def test_get_tokenized_string_normal():
+
+    text = "howdy"
+    mask = "XXX"
+    index = 1
+
+    assert _get_tokenized_string(text, mask, index) == "hXXXy"
+
+
+def test_get_tokenized_string_mask_too_long():
+
+    text = "howdy"
+    mask = "XXXXX"
+    index = 1
+
+    assert _get_tokenized_string(text, mask, index) == "hXXXX"
+
+
+def test_byte_code_to_string_no_byte_string():
+
+    fake_byte_code = "bhello"
+
+    assert _byte_code_to_string(fake_byte_code) == fake_byte_code
+
+
+def test_byte_code_to_string_start_of_header():
+
+    byte_code = "b1"
+
+    assert _byte_code_to_string(byte_code) == "\x01"
+
+
+def test_sanity_for_failed_sanity_check():
+
+    def stub_func(a, b):
+        return False
+
+    class StubFilter:
+        sanity = ["fake"]
+
+    assert sanity_test(StubFilter, "some_text", sanity_func=stub_func) == False
+
+
+def test_sanity_for_passed_sanity_checks():
+
+    def stub_func(a, b):
+        return True
+
+    class StubFilter:
+        sanity = ["sanity1", "sanity2", "sanity3"]
+
+    assert sanity_test(StubFilter, "some_text", sanity_func=stub_func)
\ No newline at end of file