Repository: HobnobMancer/cazy_webscraper (language: python)

Issue:
Conda installlation instruction correction
The `conda` installation instructions currently read:

```
[use] the following command at the command-line in the terminal: `conda asdasd cazy_webscraper`
```

which should be `conda install cazy_webscraper` (with `bioconda` in your channels) or `conda install -c bioconda cazy_webscraper`.

Bug fixing
Fixes minor bugs such as typos in the documentation, adds 'quality' checkout throughout the scraper, also adds a maximum length to the tqdm bar when scraping proteins from the protein pages.
Tidying `conda`, `pip`, and `setup.py`
1. The requirements in the three dependency lists do not currently agree with each other.

The `conda` and `pip` package managers do not overlap exactly. Some packages are available only in `pip` and others only in `conda`/`bioconda`. The repository contains an `environment.yml` for `conda` and a `requirements.txt` for `pip`. Additionally, the `setup.py` file will install `biopython` and `pandas`.

These three descriptions of code requirements should - ideally - agree, but do not:

- `setup.py` pins `biopython` at >= 1.76 and `pandas` at >= 1.0.3
- `requirements.txt` does not specify any versions
- `environment.yml` pins `biopython` at 1.78, and `pandas` at 1.1.2

If packages are to be pinned, they should really be pinned at the same versions, unless there is some specific intent otherwise (and this should be flagged). It is fine to have a specific intent (e.g. `environment.yml` specifies a very detailed known good environment, but that may be overkill, here)

2. The requirements could be organised differently - and possibly be more easily maintained.

The `setup.py` file should state which packages are necessary to run, in the `install_requires` argument, though `setuptools` does not make any attempt to ensure that the installed packages are mutually compatible. Any version pins here should perhaps take their lead from a known good configuration?

The `environment.yml` file only works with `conda`, but it does allow version pinning and also installation of `pip`-specific packages. The current `environment.yml` appears to be autogenerated (hence the very specific pins) and includes many packages that are implied dependencies of the required packages. The `environment.yml` file could be trimmed down considerably to contain only those packages that are necessary to run the script/module (and these then imply the other dependencies). Any `pip`-only modules can be specified here under a `pip` field.

The `requirements.txt` file mixes dependencies necessary for the module/script with those that are required only for development. These could be split into separate `requirements.txt` (corresponding to `install_requires` from `setup.py` and `requirements-dev.txt` files, so that packages which are not required for the module/script are not installed unnecessarily.

3. The installation instructions can then be streamlined, and separate instructions given for those who are only using, and those who are also maintaining/developing, the tool.

The four routes for a user should be something like:

- `python setup.py install` (takes dependencies from `setup.py`'s `install_requires`
- `conda env create --file environment.yml` (takes dependencies from `environment.yml` and optionally other `pip`-specific requirements files
- `conda install --file requirements.txt`
- `pip install -r requirements.txt`

then for a developer/maintainer

- `conda install --file requirements-dev.txt`
- `pip install -r requirements-dev.txt`

(then `pip install -e .`)
are good options for installing things like `flake8`/`pylint` etc.

4. Some necessary modules were missing from one or more dependency definition:

- `requests`
- `mechanicalsoup`
- `pyyaml`
- `tqdm`


Discussion / hints:

Fix patch:
diff --git a/.circleci/config.yml b/.circleci/config.yml
index 466ed6f6..c3155e03 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -15,7 +15,7 @@ jobs:
 
       - restore_cache:
           keys:
-            - cazy_wbscrpr-{{ .Branch }}-{{ checksum "requirements.txt" }}
+            - cazy_wbscrpr-{{ .Branch }}-{{ checksum "requirements.txt" }}-{{ checksum "requirements-dev.txt" }}-{{ checksum "requirements-pip.txt" }}
             - cazy_wbscrpr-
 
       - run:
@@ -24,11 +24,13 @@ jobs:
             python3 -m venv venv
             . venv/bin/activate
             pip install -r requirements.txt
+            pip install -r requirements-dev.txt
+            pip install -r requirements-pip.txt
 
       - save_cache:
           paths:
             - ./venv
-          key: cazy_wbscrpr-{{ .Branch }}-{{ checksum "requirements.txt" }}
+          key: cazy_wbscrpr-{{ .Branch }}-{{ checksum "requirements.txt" }}-{{ checksum "requirements-dev.txt" }}-{{ checksum "requirements-pip.txt" }}
 
       - run:
           name: install package
diff --git a/cazy_webscraper/__init__.py b/cazy_webscraper/__init__.py
new file mode 100644
index 00000000..7252861c
--- /dev/null
+++ b/cazy_webscraper/__init__.py
@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Author:
+# Emma E. M. Hobbs
+
+# Contact
+# eemh1@st-andrews.ac.uk
+
+# Emma E. M. Hobbs,
+# Biomolecular Sciences Building,
+# University of St Andrews,
+# North Haugh Campus,
+# St Andrews,
+# KY16 9ST
+# Scotland,
+# UK
+
+# The MIT License
+"""Web scraper to scrape the CAZy website."""
diff --git a/cazy_webscraper/cazy_webscraper.py b/cazy_webscraper/cazy_webscraper.py
new file mode 100644
index 00000000..08989e99
--- /dev/null
+++ b/cazy_webscraper/cazy_webscraper.py
@@ -0,0 +1,476 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Author:
+# Emma E. M. Hobbs
+
+# Contact
+# eemh1@st-andrews.ac.uk
+
+# Emma E. M. Hobbs,
+# Biomolecular Sciences Building,
+# University of St Andrews,
+# North Haugh Campus,
+# St Andrews,
+# KY16 9ST
+# Scotland,
+# UK
+
+# The MIT License
+"""
+Web scraper to scrape CAZy website and retrieve all protein data.
+
+:cmd_args --config: path to configruration file
+:cmd args --data_split: [None, class, family] how data is to be split/separated into dataframes
+:cmd_args --force: force overwriting content in exisiting output directory
+:cmd_args --log: path to log file, enables writing out log messages to a log file
+:cmd_args --nodelete: if true does not delete content in pre-existing output directory
+:cmd_args --output: path to output directory
+:cmd_args --subfamily: enable retrieval of subfamilies from CAZy
+:cmd_args --verbose: change logger level from warning to info, verbose logging
+
+:func main: coordinate scraping of CAZy database
+:func get_class_urls: retrieve URLs to CAZy class summary pages from CAZy homepage
+:func browser_decorator: decorator for get_page() to coordinate retrying failed connections
+:func get_page: connect to webpage and retrieval page as BeautifulSoup4 object
+
+:class Protein: A single protein from CAZy database
+:class Family: A single family from CAZy containing proteins
+"""
+
+import logging
+import re
+import sys
+import time
+
+import numpy as np
+
+from collections import defaultdict
+from typing import List, Optional
+from requests.exceptions import ConnectionError, MissingSchema
+from urllib3.exceptions import HTTPError, RequestError
+
+import mechanicalsoup
+
+from tqdm import tqdm
+
+from cazy_webscraper import crawler, file_io, parse, utilities
+
+
+class Protein:
+    """A single protein.
+
+    Each protein has a name, source organism (source), and links to external databases. The links to
+    external databases are stored in a dictionary, keyed by the external database name ('str') with
+    'list' values becuase there may be multiple links per database.
+
+    Multiple 'synonym' GenBank accession numbers maybe listed for a single protein. CAZy only
+    hyperlinks the first listed accession number. This accession is the one listed for the protein,
+    because is presumed to be the accession used by CAZy in their classification. All other listed
+    GenBank accessions are regarded as synonyms, including for example splice variants and identical
+    protein sequence submissions.
+    """
+
+    def __init__(self, name, family, ec, source, links=None, genbank_synonyms=None):
+        self.name = name
+        self.family = family
+        self.ec = ec
+        self.source = source
+        if links is None:
+            self.links = defaultdict(list)
+        else:
+            self.links = links
+        self.genbank_synonyms = genbank_synonyms
+
+    def __str__(self):
+        """Create representative string of class object"""
+        return f"{self.name} ({self.family} {self.source}): links to {self.links.keys()}"
+
+    def __repr__(self):
+        """Create representative object"""
+        return (
+            f"<Protein: {id(self)}: {self.name}, {self.family} "
+            f"({self.source}), {len(self.links)} to external databases>"
+        )
+
+    def get_protein_dict(self):
+        """Return a dictionary containing all the data of the protein."""
+        protein_dict = {"Protein_name": [self.name], "CAZy_family": [self.family]}
+
+        if len(self.ec) == 0:
+            protein_dict["EC#"] = [np.nan]
+        elif len(self.ec) == 1:
+            protein_dict["EC#"] = self.ec
+        else:
+            ec_string = "\n".join(self.ec)
+            protein_dict["EC#"] = [ec_string]
+
+        protein_dict["Source_organism"] = [self.source]
+
+        if type(self.links) is dict:
+            for database in ["GenBank", "UniProt", "PDB/3D"]:
+                try:
+                    if len(self.links[database]) == 1:
+                        protein_dict[database] = self.links[database]
+                    else:
+                        accession_string = ",\n".join(self.links[database])
+                        protein_dict[database] = [accession_string]
+                except KeyError:
+                    protein_dict[database] = [np.nan]
+        else:
+            for database in ["GenBank", "UniProt", "PDB/3D"]:
+                protein_dict[database] = [np.nan]
+        return protein_dict
+
+
+class Family:
+    """A single CAZy family."""
+
+    members = set()  # holds Protein instances
+
+    def __init__(self, name, cazy_class):
+        self.name = name
+        self.cazy_class = cazy_class
+
+    def __str__(self):
+        return f"CAZy family {self.name}: {len(self.members)} protein members"
+
+    def __repr__(self):
+        return f"<Family: {id(self)}: {self.name}, {len(self.members)} protein members"
+
+    def get_proteins(self):
+        """Return a list of all protein members of the CAZy family."""
+        return self.members
+
+    def get_family_name(self):
+        """Return family name"""
+        return self.name
+
+
+def main(argv: Optional[List[str]] = None, logger: Optional[logging.Logger] = None):
+    """Set up parser, logger and coordinate overal scrapping of CAZy.
+
+    The collected data can be stored as a singel dataframe containing (not split), split into
+    separate dataframes by class or by family. Excluded classes are CAZy classes not specified in
+    the configuration file and thus, will not be scraped. User_cazy_families is the list of CAZy
+    families specified to be scraped in the configration file.
+    """
+    # Program preparation
+    if argv is None:
+        parser = utilities.build_parser()
+        args = parser.parse_args()
+    else:
+        args = utilities.build_parser(argv).parse_args()
+
+    if logger is None:
+        logger = utilities.build_logger("cazy_webscraper", args)
+    logger.info("Run initiated")
+
+    if args.output is not sys.stdout:
+        file_io.make_output_directory(args.output, logger, args.force, args.nodelete)
+
+    if args.genbank is not None:
+        # create directory to write FASTA files to
+        if (args.genbank_output is not sys.stdout) and (args.genbank_output != args.output):
+            file_io.make_output_directory(args.genbank_output, logger, args.force, args.nodelete)
+
+    if args.pdb is not None:
+        # create directory to write structure files to
+        if (args.pdb_output is not None) and (args.pdb_output != args.output):
+            file_io.make_output_directory(args.pdb_output, logger, args.force, args.nodelete)
+
+    if args.subfamilies is True:
+        logger.warning("Enabled to retrieve subfamilies")
+
+    max_tries = args.retries + 1  # maximum number of times to try scraping a CAZy
+
+    # retrieve configuration data
+    file_io_path = file_io.__file__
+    excluded_classes, config_dict, cazy_dict = file_io.parse_configuration(
+        file_io_path,
+        args,
+        logger,
+    )
+
+    logger.info("Finished program preparation")
+    logger.info("Starting retrieval of data from CAZy")
+
+    # Crawl through and scrape CAZy website/database
+    cazy_home = "http://www.cazy.org"  # the CAZy homepage URL
+
+    # Retrieve data from CAZy database
+    get_cazy_data(cazy_home, excluded_classes, config_dict, cazy_dict, max_tries, logger, args)
+
+    logger.info(
+        (
+            "Finished scraping the CAZy website.\n"
+            "Thank you for using the cazy_webscraper.py\n"
+            "Terminating program"
+        )
+    )
+
+
+def get_cazy_data(cazy_home, excluded_classes, config_dict, cazy_dict, max_tries, logger, args):
+    """Coordinate retrieval of data from the CAZy website.
+
+    This function coordinates the crawling through the CAZy website by calling the appropriate
+    functions, and then retrieving the protein data by calling to the appropriate data again.
+
+    :param cazy_home: str, url of CAZy home page
+    :param excluded_classes: list, list of classes to not scrape from CAZy
+    :param config_dict: dict, user defined configuration of the scraper
+    :param cazy_dict: dict, dictionary of excepct CAZy synonyms for CAZy classes
+    :param max_tries: int, maximum number of times to scrape CAZy if errors are encountered
+    :param logger: logger object
+    :param args: cmd args parser
+
+    Return nothing.
+    """
+    # List of urls that were failed to be scraped
+    failed_url_scrapes = []
+
+    # retrieve links to CAZy class pages
+    # nested lists of [class_url, number_of_attempted_class_url_scrapes]
+    class_urls = get_class_urls(cazy_home, excluded_classes, max_tries, logger)
+
+    all_data = []  # stores all Family class objects if not splitting the data
+
+    logger.info("Starting retrieval of CAZy families")
+
+    # scrape each retrieved class page
+    for class_url in tqdm(class_urls, desc="Parsing CAZy classes"):
+
+        # retrieve class name from url
+        class_name = class_url[0][20:-5]
+        #  convert retrieved class name to synonym used in configuration file
+        for key in cazy_dict:
+            if class_name in cazy_dict[key]:
+                class_name = key
+
+        # retrieve URLs to families under current working CAZy class
+        # nested list of [family_url, number_of_attempts_to_scrape_family_all_page]
+        family_urls = crawler.get_cazy_family_urls(
+            class_url[0],
+            cazy_home,
+            class_name,
+            args,
+            logger,
+        )
+
+        if family_urls is None:  # couldn't retrieve URLs to family for working CAZy class
+            # add one to the number of scrapping attempts
+            class_url[1] += 1
+
+            if class_url[1] == max_tries:  # max number of attempts to scrape has been met
+                failed_url_scrapes += (
+                    f"{class_url[0]} - no CAZy familes from this class were scraped"
+                )
+                continue
+            else:
+                class_urls += class_url  # retry scraping Class page after the other Class URLs
+                continue
+
+        families = []  # store Family class objects if splitting data be class
+
+        logger.info("Starting retrieval of protein records of protein records from families")
+
+        # Scrape the familes of the current CAZy class, retrieving protein data
+
+        if (config_dict is None) or (config_dict[class_name] is None):
+            # No (sub)families were specified, therefore, scraping all families of the CAZy class
+
+            for family_url in tqdm(family_urls, desc="Parsing CAZy families"):
+                # check url format is correct
+                try:
+                    re.match(
+                        r"http://www.cazy.org/(\D{2,3})(\d+|\d+_\d+).html", family_url[0]
+                    ).group()
+                except AttributeError:
+                    logger.warning(
+                        f"Formate of URL {family_url[0]} is incorrect.\n"
+                        "Will not attempt to scrape this URL."
+                    )
+                    failed_url_scrapes += f"{family_url[0]} - url format was incorrect"
+                    continue
+
+                family = None
+                family_name = family_url[0][(len(cazy_home) + 1): -5]
+                # build family object, populated by Proteins catalogued under the CAZy family
+                family = crawler.parse_family(family_url[0], family_name, cazy_home, logger)
+
+                # [family_object, error]
+                if family[1] is not None:  # Scraping family '_all' page was unsuccessful
+                    # add one to the number of attempted scrapes the CAZy family's '_all' page
+                    family_url[1] += 1
+
+                    if family_url[1] == max_tries:  # max number of scraping attempts reached
+                        failed_url_scrapes += (
+                            f"{family_url[0]} - the following error was raised {family[2]}"
+                        )
+                        continue
+                    else:
+                        family_urls += family_url
+                        continue
+
+                # store the family if scraped successfully
+                if args.data_split == "family":
+                    logger.info(f"Data split by Family. Writing out df for {family_name}")
+                    parse.proteins_to_dataframe([family[0]], args, logger)
+                else:
+                    families.append(family[0])
+
+        else:
+            # scrape only (sub)families specified in the config file
+
+            for family_url in tqdm(family_urls, desc="Parsing CAZy families"):
+                # check url format is correct
+                try:
+                    re.match(
+                        r"http://www.cazy.org/(\D{2,3})(\d+|\d+_\d+).html", family_url[0]
+                    ).group()
+                except AttributeError:
+                    logger.warning(
+                        (
+                            f"Formate of URL {family_url[0]} is incorrect.\n"
+                            "Will not attempt to scrape this URL."
+                        )
+                    )
+                    continue
+
+                family = None
+                family_name = family_url[0][(len(cazy_home) + 1) : -5]
+
+                # Allows retrieval of subfamilies when only the parent CAZy family was named in the
+                # config file
+                if (args.subfamilies is True) and (family_name.find("_") != -1):
+                    name_check = family_name[: (family_name.find("_"))]
+                else:
+                    name_check = family_name
+
+                if name_check in config_dict[class_name]:
+                    # build family object, populated by Proteins catalogued under the CAZy family
+                    family = crawler.parse_family(family_url[0], family_name, cazy_home, logger)
+
+                    # [family_object, error]
+                    if family[1] is not None:  # Scraping family '_all' page was unsuccessful
+                        # add one to the number of attempted scrapes the CAZy family's '_all' page
+                        family_url[1] += 1
+
+                        if family_url[1] == max_tries:  # max number of scraping attempts reached
+                            failed_url_scrapes += (
+                                f"{family_url[0]} - the following error was raised {family[2]}"
+                            )
+                            continue
+                        else:
+                            family_urls += family_url
+                            continue
+
+                    # store the family if scraped successfully
+                    if args.data_split == "family":
+                        logger.info(f"Data split by Family. Writing out df for {family_name}")
+                        parse.proteins_to_dataframe([family[0]], args, logger)
+                    else:
+                        families.append(family[0])
+
+        if args.data_split == "class":
+            if len(families) != 0:
+                logger.info(f"Data split by Class. Writing out df for {class_name}")
+                parse.proteins_to_dataframe(families, args, logger)
+            else:
+                logger.warning(f"Didn't retrieve any families for {class_name}")
+
+        elif args.data_split is None:
+            all_data += families
+
+    if args.data_split is None:
+        if len(all_data) != 0:
+            logger.info("Data was not split. Writing all retrieved data to a single df")
+            parse.proteins_to_dataframe(all_data, args, logger)
+        else:
+            logger.warning("Didn't retrieve any protein data from CAZy")
+
+    # write out URLs which failed to be scaped
+    file_io.write_out_failed_scrapes(failed_url_scrapes, args, logger)
+
+    return
+
+
+def get_class_urls(cazy_home, excluded_classes, max_tries, logger):
+    """Retrieve class urls, add storage of number of attempted scrapes of each class URL.
+
+    :param cazy_url: str, URL to the CAZy home page.
+    :param excluded_classes: list, list of CAZy classes not to be scraped
+    :param max_tries: int, maximum number of times to try scrape if errors are encountered
+    :param logger: logger object
+
+    Return list of CAZy class URLs.
+    """
+
+    class_urls = crawler.get_cazy_class_urls(cazy_home, excluded_classes, max_tries, logger)
+
+    try:
+        if len(class_urls) == 0:
+            logger.error("Failed to retrieve URLs to CAZy class pages.\nTerminating program")
+            sys.exit(1)
+    except TypeError:  # rased when class_pages is None
+        logger.error("Failed to retrieve URLs to CAZy class pages.\nTerminating program")
+        sys.exit(1)
+
+    # add storing the number of times an attempt to scrape the class page as been performed
+    index = 0
+    for index in range(len(class_urls)):
+        # list structure [class_url, number_of_tried_scrapes]
+        class_urls[index] = [class_urls[index], 0]
+
+    return class_urls
+
+
+def browser_decorator(func):
+    """Decorator to retry the wrapped function up to 'retries' times."""
+
+    def wrapper(*args, retries=10, **kwargs):
+        tries, success, err = 0, False, None
+        while not success and (tries < retries):
+            try:
+                response = func(*args, **kwargs)
+            except (
+                ConnectionError,
+                HTTPError,
+                OSError,
+                MissingSchema,
+                RequestError,
+            ) as err_message:
+                success = False
+                response = None
+                err = err_message
+            if response is not None:  # response was successful
+                success = True
+            # if response from webpage was not successful
+            tries += 1
+            time.sleep(10)
+        if (not success) or (response is None):
+            return [None, err]
+        else:
+            return [response, None]
+
+    return wrapper
+
+
+@browser_decorator
+def get_page(url):
+    """Create browser and use browser to retrieve page for given URL.
+
+    :param url: str, url to webpage
+
+    Return browser response object (the page).
+    """
+    # create browser object
+    browser = mechanicalsoup.Browser()
+    # create response object
+    page = browser.get(url)
+    page = page.soup
+
+    return page
+
+
+if __name__ == "__main__":
+    main()
diff --git a/cazy_webscraper/crawler/__init__.py b/cazy_webscraper/crawler/__init__.py
new file mode 100644
index 00000000..e0dc41da
--- /dev/null
+++ b/cazy_webscraper/crawler/__init__.py
@@ -0,0 +1,382 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Author:
+# Emma E. M. Hobbs
+
+# Contact
+# eemh1@st-andrews.ac.uk
+
+# Emma E. M. Hobbs,
+# Biomolecular Sciences Building,
+# University of St Andrews,
+# North Haugh Campus,
+# St Andrews,
+# KY16 9ST
+# Scotland,
+# UK
+
+# The MIT License
+"""Module for crawling through the CAZy website and parsing HTML from the CAZy website.
+
+:func get_cazy_family_urls: retrieve URLs to families on CAZy class summary page
+:func get_subfamily_links: retrieve URLs to subfamilies on CAZy class summary page
+:func parse_family: build Family class object to represent CAZy family
+:func parse_family_pages: retrieve all URLs to pages containing protein records for CAZy family
+:func parse_proteins: retrieve protein records from protein table page
+:func row_to_protein: parse the protein record to build a Protein class object
+"""
+
+
+import re
+import sys
+import time
+
+import numpy as np
+
+from tqdm import tqdm
+
+from cazy_webscraper.cazy_webscraper import get_page, Family, Protein
+
+
+def get_cazy_class_urls(cazy_home, excluded_classes, max_tries, logger):
+    """Returns a list of CAZy class main/home page URLs for each specified class as the CAZy site.
+
+    :param cazy_url: str, URL to the CAZy home page.
+    :param excluded_classes: list, list of CAZy classes not to be scraped
+    :param max_tries: int, maximum number of times to try scrape if errors are encountered
+    :param logger: logger object
+
+    Return list of URLs.
+    """
+    logger.info("Retrieving URLs to summary CAZy class pages")
+
+    # define items to be excluded from returned class list, ALWAYS exlide links to genomes
+    if excluded_classes is not None:
+        exclusions = tuple(["<strong>Genomes</strong>"] + excluded_classes)
+    else:
+        exclusions = "<strong>Genomes</strong>"
+
+    home_page = [None, None]
+    tries = 0  # number of the try/attempt
+
+    while (home_page[0] is None) and (tries < max_tries):
+        home_page = get_page(cazy_home)
+
+        if (home_page[0] is None) and (tries < max_tries):
+            logger.error(
+                f"Failed to connect to CAZy homepage after 10 attempts,\n"
+                f"for attempt# {(tries+1)}/{max_tries}\n"
+                "The following error was raised:\n"
+                f"{home_page[1]}\n"
+                f"Reattempting for attempt# {(tries+2)} in 10s."
+            )
+            time.sleep(10)
+            tries += 1
+
+    if home_page[0] is None:
+        logger.error(
+            (
+                "Failed to connect to CAZy home-page after multiple attempts.\n"
+                "The following error was raised:\n"
+                f"{home_page[1]}"
+                "Could not retrieve URLs to CAZy classes.\n"
+                "Check the network connection.\n"
+                "Terminating program."
+            )
+        )
+        sys.exit(1)
+
+    try:
+        return [
+            f"{cazy_home}/{_['href']}"
+            for _ in home_page[0].find_all("a", {"class": "spip_out"})
+            if (not _["href"].startswith("http")) and (str(_.contents[0]) not in exclusions)
+        ]
+    except AttributeError:  # raise if can't find results with find_all("a", {"class": "spip_out"})
+        return None
+
+
+def get_cazy_family_urls(class_url, cazy_home, class_name, args, logger):
+    """Retrieve all protein members of each CAZy family within the given CAZy class.
+
+    :param class_url: str, URL to the CAZy class
+    :param cazy_home: str, URL to CAZy home page
+    :param class_name: str, name of CAZy class
+    :param args: args parser object
+    :param logger: logger object
+
+    Returns list of URLs to family pages.
+    """
+    logger.info(f"Retrieving URLs to families under {class_name}")
+
+    # scrape the class page
+    class_page = get_page(class_url)
+
+    if class_page[0] is None:
+        logger.error(
+            (
+                f"Failed to connect to {class_url} after 10 attempts.\n"
+                "The following error was raised:\n"
+                f"{class_page[1]}"
+                "Could not retrieve URLs to CAZy famileis for this class.\n"
+                "This class will be skipped during the scraping process."
+            )
+        )
+        return None
+
+    # retrieve the <h3> element that titles the div section containing the tables of family links
+    family_h3_element = [
+        _
+        for _ in class_page[0].find_all("h3", {"class": "spip"})
+        if str(_.contents[0]).strip() == "Tables for Direct Access"
+    ][0]
+
+    # retrieve all tables within the parent div section of the <h3> element
+    tables = family_h3_element.parent.find_all("table")
+
+    # tables[0] is the table containing links to CAZy families
+    # tables[1] is the table containing the link to unclassified proteins
+
+    family_urls = family_urls = [f"{cazy_home}/{_['href']}" for _ in tables[0].find_all("a")]
+    family_urls.append(f"{cazy_home}/{tables[1].a['href']}")
+    if args.subfamilies is True:
+        subfam_urls = get_subfamily_links(family_h3_element, cazy_home, logger)
+        if subfam_urls is not None:
+            family_urls += subfam_urls
+
+    if family_urls is not None:
+        # add storage of number of attempts to scrape the Family _all page
+        index = 0
+        for index in range(len(family_urls)):
+            family_urls[index] = [family_urls[index], 0]
+
+    return family_urls
+
+
+def get_subfamily_links(family_h3_element, cazy_home, logger):
+    """Retrieve URL links to CAZy subfamilies.
+
+    :param family_h3_element: bs4.element.Tag, h3 element titling the page div
+    :param cazy_home: str, URL to CAZy home_page
+    :param logger: logger object
+
+    Return list of URLs to subfamilies.
+    """
+    parent_div = family_h3_element.parent
+    all_links = parent_div.find_all("a")
+
+    pattern = re.compile(r"\D+?\d+?_\d+?\.html")
+
+    urls = []  # empty list to store subfamily URLs
+
+    for link in all_links:
+        try:
+            search_result = re.search(pattern, link["href"])
+            urls.append(f"{cazy_home}/{search_result.group()}")
+        except (KeyError, AttributeError) as error:
+            # KeyError raised if link does not have ['href']
+            # AttributeError error raised if search_result is None becuase not subfam link
+            pass
+
+    if len(urls) == 0:
+        return
+    else:
+        return urls
+
+
+def parse_family(family_url, family_name, cazy_home, logger):
+    """Returns a Family object with Protein members, scraped from CAZy.
+
+    :param family_url: str, URL to CAZy family summary page
+    :param family_name: str, name of CAZy family
+    :param cazy_home: str, URL to CAZy home page
+    :param logger: logger object
+
+    Return list of Family object encapsulating data from the passed family
+    summary page, and an error code if failed to scrape the family 'all' page.
+    """
+    logger.info(f"Starting retrieval of proteins for {family_name} from {family_url}")
+
+    # retrieve class from family name
+    pattern = re.compile(r"\D+")
+    search_result = re.match(pattern, family_name)
+    cazy_class = search_result.group()
+
+    family = Family(family_name, cazy_class)
+    family.members = set()
+
+    # compile URL to first family page of protein records
+    first_pagination_url = family_url.replace(".html", "_all.html")
+
+    # check url formating
+    try:
+        re.match(
+            r"http://www.cazy.org/\D{2,3}(\d+|\d+_\d+)_all.html", first_pagination_url
+        ).group()
+    except AttributeError:
+        logger.warning(
+            f"Incorrect formatting of first protein table page URL: {first_pagination_url}\n"
+            "Will not try and connect to this URL."
+        )
+        return [family, "Incorrect first protein table page URL formating"]
+
+    first_pagination_page = get_page(first_pagination_url)
+
+    if first_pagination_page[0] is None:
+        logger.warning(
+            (
+                f"Could not connect to {family_url}\n"
+                "The following error was raised:\n"
+                f"{first_pagination_page[1]}"
+                f"No protein records for CAZy family {family_name} will be retried."
+            )
+        )
+        return [family, first_pagination_page[1]]
+
+    protein_page_urls = get_protein_page_urls(
+        first_pagination_url,
+        first_pagination_page[0],
+        cazy_home,
+    )
+
+    for protein in tqdm(
+        (y for x in (parse_proteins(url, family_name, logger) for url in protein_page_urls) for y in x),
+        total=len(protein_page_urls),
+        desc=f"Scraping protein pages for {family_name}",
+    ):
+        family.members.add(protein)
+
+    return [family, None]
+
+
+def get_protein_page_urls(first_pagination_url, first_pagination_page, cazy_home):
+    """Retrieve the URLs to all pages containing proteins for the current working family.
+
+    :param first_pagination_url: str, URL to first page contaiing proteins
+    :param first_pagination_page: BS4 object, first page containing proteins
+    :param cazy_home: str, URL of CAZy homepage
+
+    Return list of URLs.
+    """
+    protein_page_urls = [first_pagination_url]
+
+    # retrieve the URL to the final page of protein records in the pagination listing
+    try:
+        last_pagination_url = first_pagination_page.find_all("a", {"class": "lien_pagination", "rel": "nofollow"})[-1]
+    except IndexError:  # there is no pagination; a single-query entry
+        last_pagination_url = None
+
+    if last_pagination_url is not None:
+        url_prefix = last_pagination_url["href"].split("PRINC=")[0] + "PRINC="
+        last_princ_no = int(last_pagination_url["href"].split("PRINC=")[-1].split("#pagination")[0])
+        url_suffix = "#pagination" + last_pagination_url["href"].split("#pagination")[-1]
+
+        # Build list of urls to all pages in the pagination listing, increasing the PRINC increment
+        protein_page_urls.extend(
+            [f"{cazy_home}/{url_prefix}{_}{url_suffix}" for _ in range(1000, last_princ_no + 1000, 1000)]
+        )
+
+    return protein_page_urls
+
+
+def parse_proteins(protein_page_url, family_name, logger):
+    """Returns generator of Protein objects for all protein rows on a single CAZy family page.
+
+    :param protein_page_url, str, URL to the CAZy family page containing protein records
+    :param family_name: str, name of CAZy family
+    :param logger: logger object
+
+    Return generator object.
+    """
+    logger.info(f"Retrieving proteins from {protein_page_url}")
+
+    protein_page = get_page(protein_page_url)
+
+    if protein_page[0] is None:
+        logger.warning(
+            (
+                f"Could not connect to {protein_page_url}\n"
+                "The following error was raised:\n"
+                f"{protein_page[1]}\n"
+                f"No protein records from this page will be retried."
+            )
+        )
+        return
+
+    # retrieve protein record table
+    protein_table = protein_page[0].find_all("table", {"class": "listing"})[0]
+    protein_rows = [
+        _ for _ in protein_table.descendants if (_.name == "tr") and ("id" not in _.attrs) and ("class" not in _.attrs)
+    ]
+
+    for row in protein_rows:
+        yield row_to_protein(row, family_name)
+
+
+def row_to_protein(row, family_name):
+    """Returns a Protein object representing a single protein row from a CAZy family protein page.
+
+    Each row, in order, contains the protein name, EC number, source organism, GenBank ID(s),
+    UniProt ID(s), and PDB accession(s).
+
+    :param row: tr element from CAZy family protein page
+    :param family_name: str, name of CAZy family
+
+    Return Protein instance.
+    """
+    # retrieve list of cells ('td' elements) in row
+    tds = list(row.find_all("td"))
+
+    protein_name = tds[0].contents[0].strip()
+    source_organism = tds[2].a.get_text()
+    ec_numbers = []
+    links = {}
+
+    all_links = None
+    try:
+        all_links = tds[1].find_all("a")
+    except TypeError:  # raised when no EC numbers are listed
+        pass
+
+    if all_links is not None:
+        for link in all_links:
+            ec_numbers.append(link.text)
+    else:
+        ec_numbers = [np.nan]
+
+    # test for len(tds[x].contents) in case there is no link,
+    # the check of .name then ensures link is captured
+    if len(tds[3].contents) and tds[3].contents[0].name == "a":
+        links["GenBank"] = [f"{_.get_text()} {_['href']}" for _ in tds[3].contents if _.name == "a"]
+    if len(tds[4].contents) and tds[4].contents[0].name == "a":
+        links["UniProt"] = [f"{_.get_text()} {_['href']}" for _ in tds[4].contents if _.name == "a"]
+    if len(tds[5].contents) and tds[5].contents[0].name == "a":
+        links["PDB/3D"] = [f"{_.get_text()} {_['href']}" for _ in tds[5].contents if _.name == "a"]
+
+    # Retrieve GenBank accession synonms
+    try:
+        if len(links["GenBank"]) != 0:
+            # retrieve HTML link GenBank accession (CAZy only hyperlinks the first listed accession)
+            key = links["GenBank"][0]
+            key = key.split(" ")[0]  # remove the URL address
+
+            # retrieve all GenBank accessions listed in td element
+            all_genbank_accessions = tds[3].get_text(separator=" ")
+            all_genbank_accessions = all_genbank_accessions.split(" ")
+
+            if key in all_genbank_accessions:
+                all_genbank_accessions.remove(key)
+
+            # remove duplicates
+            all_genbank_accessions = list(dict.fromkeys(all_genbank_accessions))
+
+            if len(all_genbank_accessions) == 0:
+                genbank_synonyms = None
+            else:
+                genbank_synonyms = {key: all_genbank_accessions}
+        else:
+            genbank_synonyms = None
+    except KeyError:
+        genbank_synonyms = None
+
+    return Protein(protein_name, family_name, ec_numbers, source_organism, links, genbank_synonyms)
diff --git a/cazy_webscraper/file_io/__init__.py b/cazy_webscraper/file_io/__init__.py
new file mode 100644
index 00000000..985ed37c
--- /dev/null
+++ b/cazy_webscraper/file_io/__init__.py
@@ -0,0 +1,230 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Author:
+# Emma E. M. Hobbs
+
+# Contact
+# eemh1@st-andrews.ac.uk
+
+# Emma E. M. Hobbs,
+# Biomolecular Sciences Building,
+# University of St Andrews,
+# North Haugh Campus,
+# St Andrews,
+# KY16 9ST
+# Scotland,
+# UK
+
+# The MIT License
+"""Module for handling input and output files and directories."""
+
+import json
+import shutil
+import sys
+import yaml
+
+from pathlib import Path
+
+
+def make_output_directory(output, logger, force, nodelete):
+    """Create output directory for genomic files.
+
+    :param output: path, path of dir to be created
+    :param logger: logger object
+    :param force: bool, enable/disable creating dir if already exists
+    :param nodelete: bool, enable/disable deleting content in existing dir
+
+    Raises FileExistsError if an attempt is made to create a directory that already
+    exist and force (force overwrite) is False.
+
+    Return Nothing
+    """
+    if force is True:
+        logger.warning(
+            "Output directory %s exists, nodelete is %s", output, nodelete,
+        )
+        if nodelete and output.exists():
+            logger.warning("Not deleting directory %s", output)
+        elif output.exists():
+            logger.warning("Deleting directory %s", output)
+            shutil.rmtree(output)
+
+    logger.info("Creating directory %s", output)
+    try:
+        output.mkdir(exist_ok=force)
+    except FileExistsError:
+        logger.warning(
+            (
+                "Out directory already exists."
+                "New directory not made, writing to existing directory."
+            )
+        )
+
+
+def parse_configuration(file_io_path, args, logger):
+    """Parse configuration data, and retrieve user specified CAZy classes and families.
+
+    Return only the CAZy class synonoms dictionary if a path to a configuration file was not given.
+    This results in the default behaviour of the webscraper to scrape the entirty of CAZy to be
+    invoked. If no items are listed under a heading/tag in the config file, the retrieved item will
+    be a None type object.
+
+    :param file_io_path: str, path to directory where file_io is installed
+    :param args: parser arguments
+    :param logger: logger object
+
+    Return list of classes not to scrape, dict of families to scrape, and dict of class synonoms.
+    """
+    # open dictionary of accepted CAZy class synonyms)
+    dict_path = file_io_path.replace("__init__.py", "cazy_dictionary.json")
+    dict_path = Path(dict_path)
+
+    try:
+        with open(dict_path, "r") as fh:
+            cazy_dict = json.load(fh)
+            # create list of standardised CAZy classes
+            std_class_names = list(cazy_dict.keys())
+    except FileNotFoundError:
+        logger.error(
+            "Could not open the CAZy synonym dictionary\n"
+            "Terminating programme"
+        )
+        sys.exit(1)
+
+    if args.config is None:
+        return None, None, cazy_dict
+
+    try:
+        # open configuration file
+        with open(args.config) as fh:
+            config_dict = yaml.full_load(fh)
+    except FileNotFoundError:
+        logger.warning(
+            "Could not find configuration file.\n"
+            "Make sure path to the configuration file is correct\n"
+            "Scrapping will continue using default scraper configuration\n"
+        )
+        return None, None, cazy_dict
+
+    # Retrieve CAZy classes listed in config file
+    try:
+        cazy_classes = config_dict["classes"]
+    except (KeyError, TypeError) as e:
+        logger.warning(
+            (
+                "Did not find the 'classes' tag in the configuration files.\n"
+                "Did not retrieve any CAZy classes from configuration file."
+            )
+        )
+        cazy_classes = []
+
+    if cazy_classes is None:
+        logger.info("Did not retrieve any items under 'classes' in the config file.")
+        cazy_classes = []
+
+    if len(cazy_classes) != 0:
+        # standardise CAZy class names
+        cazy_classes = parse_user_cazy_classes(cazy_classes, cazy_dict, std_class_names, logger)
+
+    # retrieve classes of families/subfamilies specified in configuration file
+    for key in config_dict:
+        if (key != "classes") and (key not in cazy_classes) and (config_dict[key] is not None):
+            # add the class of families to be scraped to the list of CAZy classes to be scraped
+            cazy_classes.append(key)
+
+    # create list of CAZy classes not to be scraped
+    excluded_classes = std_class_names
+    excluded_classes = list(set(excluded_classes).difference(cazy_classes))
+
+    if len(excluded_classes) != 0:
+        # change names of classes into format for excluding classes during scrape
+        index = 0
+        for index in range(len(excluded_classes)):
+            excluded_classes[index] = f"<strong>{excluded_classes[index]}</strong>"
+    else:
+        excluded_classes = None
+
+    return excluded_classes, config_dict, cazy_dict
+
+
+def parse_user_cazy_classes(cazy_classes, cazy_dict, std_class_names, logger):
+    """Standardise the CAZy class names listed in configuration file.
+
+    :param cazy_classes: list, list of CAZy classes from configuration file
+    :param cazy_dict: dict, keyed by class name, keyed by list of accepted synonoms
+    :param std_class_names: list, list of all CAZy classes
+    :param logger: logger object
+
+    Return list of CAZy classes listed by user in the configuration file.
+    """
+    logger.info("Standardising names of class listed in configuration file")
+
+    index = 0
+    for index in range(len(cazy_classes)):
+        # identify user defined CAZy classes not written in standardised format
+        if cazy_classes[index] not in std_class_names:
+            for key in cazy_dict:
+                if cazy_classes[index] in cazy_dict[key]:  # if in synonyms in cazy_dict
+                    cazy_classes[index] = key  # standardise the class name
+
+        # check all names are standardised, remove names that could not be standardised
+        if cazy_classes[index] not in std_class_names:
+            logger.warning(
+                (
+                    f"'{cazy_classes[index]}' could not be standardised.\n"
+                    "Please use a synonym in the file_io/cazy_dictionary.json.\n"
+                    f"'{cazy_classes[index]}' will NOT be scraped."
+                )
+            )
+            del cazy_classes[index]
+
+    return cazy_classes
+
+
+def write_out_df(dataframe, df_name, outdir, logger, force):
+    """Write out dataframe to output directory.
+
+    :param dataframe: pandas dataframe
+    :param df_name: str, name of dataframe
+    :param outdir: Path, path to output directory
+    :param logger: logger object
+    :param force: bool, enable/disable over writing of existing file
+
+    Return nothing.
+    """
+    # build output path
+    output_path = outdir / f"{df_name}.csv"
+
+    logger.info("Checking if output directory for dataframe already exists")
+    if output_path.exists():
+        if force is False:
+            logger.warning(
+                (
+                    "Specified directory for dataframe already exists.\n"
+                    "Exiting writing out dataframe."
+                )
+            )
+            return ()
+        else:
+            logger.warning(
+                (
+                    "Specified directory for dataframe already exists.\n"
+                    "Forced overwritting enabled."
+                )
+            )
+
+    logger.info("Writing out species dataframe to directory")
+    dataframe.to_csv(output_path)
+    return
+
+
+def write_out_failed_scrapes(failed_urls, args, logger):
+    """Write out the URLs for which a connection to CAZy failed.
+
+    :param failed_urls: list of URLs
+    :param args: cmd args parser
+    :param logger: logger object
+
+    Return nothing.
+    """
+    return
\ No newline at end of file
diff --git a/cazy_webscraper/file_io/cazy_dictionary.json b/cazy_webscraper/file_io/cazy_dictionary.json
new file mode 100644
index 00000000..b0850298
--- /dev/null
+++ b/cazy_webscraper/file_io/cazy_dictionary.json
@@ -0,0 +1,8 @@
+{
+    "Glycoside Hydrolases (GHs)": ["Glycoside-Hydrolases", "Glycoside-Hydrolases", "Glycoside_Hydrolases", "GlycosideHydrolases", "GLYCOSIDE-HYDROLASES", "GLYCOSIDE-HYDROLASES", "GLYCOSIDE_HYDROLASES", "GLYCOSIDEHYDROLASES", "glycoside-hydrolases", "glycoside-hydrolases", "glycoside_hydrolases", "glycosidehydrolases", "GH", "gh"],
+    "GlycosylTransferases (GTs)": ["Glycosyl-Transferases", "GlycosylTransferases", "Glycosyl_Transferases", "Glycosyl Transferases", "GLYCOSYL-TRANSFERASES", "GLYCOSYLTRANSFERASES", "GLYCOSYL_TRANSFERASES", "GLYCOSYL TRANSFERASES", "glycosyl-transferases", "glycosyltransferases", "glycosyl_transferases", "glycosyl transferases", "GT", "gt"],
+    "Polysaccharide Lyases (PLs)": ["Polysaccharide Lyases", "Polysaccharide-Lyases", "Polysaccharide_Lyases", "PolysaccharideLyases", "POLYSACCHARIDE LYASES", "POLYSACCHARIDE-LYASES", "POLYSACCHARIDE_LYASES", "POLYSACCHARIDELYASES", "polysaccharide lyases", "polysaccharide-lyases", "polysaccharide_lyases", "polysaccharidelyases", "PL", "pl"],
+    "Carbohydrate Esterases (CEs)": ["Carbohydrate Esterases", "Carbohydrate-Esterases", "Carbohydrate_Esterases", "CarbohydrateEsterases", "CARBOHYDRATE ESTERASES", "CARBOHYDRATE-ESTERASES", "CARBOHYDRATE_ESTERASES", "CARBOHYDRATEESTERASES", "carbohydrate esterases", "carbohydrate-esterases", "carbohydrate_esterases", "carbohydrateesterases", "CE", "ce"],
+    "Auxiliary Activities (AAs)": ["Auxiliary Activities", "Auxiliary-Activities", "Auxiliary_Activities", "AuxiliaryActivities", "AUXILIARY ACTIVITIES", "AUXILIARY-ACTIVITIES", "AUXILIARY_ACTIVITIES", "AUXILIARYACTIVITIES", "auxiliary activities", "auxiliary-activities", "auxiliary_activities", "auxiliaryactivities", "AA", "aa"],
+    "Carbohydrate-Binding Modules (CBMs)": ["Carbohydrate-Binding-Modules", "Carbohydrate_Binding_Modules", "Carbohydrate_Binding Modules", "CarbohydrateBindingModules", "CARBOHYDRATE-BINDING-MODULES", "CARBOHYDRATE_BINDING_MODULES", "CARBOHYDRATE_BINDING MODULES", "CARBOHYDRATEBINDINGMODULES", "carbohydrate-binding-modules", "carbohydrate_binding_modules", "carbohydrate_binding modules", "carbohydratebindingmodules", "CBMs", "CBM", "cbms", "cbm"]
+}
\ No newline at end of file
diff --git a/cazy_webscraper/parse/__init__.py b/cazy_webscraper/parse/__init__.py
new file mode 100644
index 00000000..7b03f7d2
--- /dev/null
+++ b/cazy_webscraper/parse/__init__.py
@@ -0,0 +1,422 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Author:
+# Emma E. M. Hobbs
+
+# Contact
+# eemh1@st-andrews.ac.uk
+
+# Emma E. M. Hobbs,
+# Biomolecular Sciences Building,
+# University of St Andrews,
+# North Haugh Campus,
+# St Andrews,
+# KY16 9ST
+# Scotland,
+# UK
+
+# The MIT License
+"""Module for parsing the webpages from CAZy."""
+
+
+import json
+import os
+import re
+import sys
+import time
+
+import pandas as pd
+
+from datetime import datetime
+
+from Bio import Entrez, SeqIO
+from Bio.PDB import PDBList
+from tqdm import tqdm
+
+from scraper.file_io import write_out_df
+
+
+def proteins_to_dataframe(families, args, logger):
+    """Write protein members of CAZy families to a pandas dataframe.
+
+    Duplicate rows are removed. These rows must be identical across the entire row, not only share
+    the same protein name. Duplicates are mostly likely to arise when scraping families and
+    subfamilies becuase proteins under the subfamily are also catalogued under the parent family.
+
+    :param families: list, list of CAZy families containing protein members
+    :param args: args parser
+    :param logger: logger object
+
+    Return nothing.
+    """
+    # Build empty dictionary to store GenBank accession number synonyms
+    all_genbank_synonyms = {}
+
+    # Build empty dataframe to add proteins to
+    protein_dataframe = pd.DataFrame(
+        data={},
+        columns=[
+            "Protein_name",
+            "CAZy_family",
+            "EC#",
+            "Source_organism",
+            "GenBank",
+            "UniProt",
+            "PDB/3D",
+        ]
+    )
+
+    # add proteins to dataframe
+    for family in families:
+        proteins = family.get_proteins()
+        for protein in tqdm(proteins, desc=f"Writing {family.name} proteins to df"):
+            if protein is not None:
+                # write out protein to df
+                protein_dict = protein.get_protein_dict()
+                df = pd.DataFrame(protein_dict)
+                protein_dataframe = protein_dataframe.append(df, ignore_index=True)
+                # add synonym GenBank accessions to all synonyms accessions
+                genbank_synonyms = protein.genbank_synonyms
+                if genbank_synonyms is not None:
+                    all_genbank_synonyms.update(genbank_synonyms)
+
+    time_stamp = datetime.now().strftime("%Y-%m-%d--%H-%M-%S")
+
+    if args.subfamilies is True:
+        subfam = "_incld_subfams"
+    else:
+        subfam = ""
+    # compile dataframe name
+    if args.data_split == "family":
+        df_name = f"cazy_{families[0].name}{subfam}_{time_stamp}"
+        synonyms_dict = f"cazy_genbank_synonyms_{families[0].name}{subfam}_{time_stamp}"
+    elif args.data_split == "class":
+        df_name = f"cazy_{families[0].cazy_class}{subfam}_{time_stamp}"
+        synonyms_dict = f"cazy_genbank_synonyms_{families[0].cazy_class}{subfam}_{time_stamp}"
+    else:
+        df_name = f"cazy_scrape{subfam}_{time_stamp}"
+        synonyms_dict = f"cazy_genbank_synonyms{subfam}_{time_stamp}"
+
+    # Remove duplicate proteins
+    # This can arise when scraping subfamilies and families within a class
+    # The proteins within the subfamilies will also be listed under their parent family
+    protein_dataframe = protein_dataframe.drop_duplicates()
+
+    # write out dataframe
+    write_out_df(protein_dataframe, df_name, args.output, logger, args.force)
+
+    # write out dictionary of genbank_synonyms
+    if args.output is sys.stdout:
+        json.dump(all_genbank_synonyms, sys.stdout)
+
+    else:
+        output_path = args.output / f"{synonyms_dict}.json"
+        with open(output_path, "w") as fh:
+            json.dump(all_genbank_synonyms, fh)
+
+    # Additional parsing and retrieval of data: protein sequences and/or structures
+    if (args.genbank is not None) and (args.pdb is not None):
+        logger.info(f"Retrieving structures and sequences, for proteins in {df_name}")
+        get_structures_and_sequences(protein_dataframe, df_name, args, logger)
+
+    elif (args.genbank is not None) and (args.pdb is None):
+        logger.info(f"Retrieving FASTA files from GenBank, for proteins in {df_name}")
+        get_genbank_fasta(protein_dataframe, df_name, args, logger)
+
+    elif (args.genbank is None) and (args.pdb is not None):
+        logger.info(f"Retrieve PDB structure files for proteins in {df_name}")
+        get_pdb_structures(protein_dataframe, df_name, args, logger)
+
+    return
+
+
+def get_structures_and_sequences(protein_dataframe, df_name, args, logger):
+    """Coordinate the retrieval of CAZyme structure and sequences from PDB and GenBank.
+
+    :param protein_dataframe: Pandas dataframe, dataframe containing protein data from CAZy
+    :param df_name: str, name of the CAZy dataframe
+    :param args: cmd args parser
+    :param logger: logger object
+
+    Return nothing.
+    """
+    # set user email address for Entrez
+    Entrez.email = args.genbank
+
+    index = 0
+    pdbl = PDBList()
+    for index in tqdm(
+        range(len(protein_dataframe["Protein_name"])),
+        desc=f"Downloading {df_name} PDBs and FASTAs",
+    ):
+        # Retrieve accession from GenBank cell
+        df_row = protein_dataframe.iloc[index]
+        genbank_accession, cazy_family = get_genbank_accession(df_row, df_name, index, logger)
+        pdb_accessions = get_pdb_accessions(df_row, df_name, logger)
+
+        if genbank_accession is not None:
+            # get FASTA file from GenBank
+            fasta_name = f"{genbank_accession}_{cazy_family}.fasta"
+
+            if args.genbank_output is not sys.stdout:
+                fasta_name = args.genbank_output / fasta_name
+
+            download_fasta(genbank_accession, fasta_name, args, logger)
+
+        if pdb_accessions is not None:
+            # Get structure from PDB
+            if args.pdb_output is not None:
+                for accession in pdb_accessions:
+                    pdbl.retrieve_pdb_file(
+                        f"{accession}",
+                        file_format=args.pdb,
+                        pdir=args.pdb_output,
+                    )
+
+            else:
+                if args.output is not sys.stdout:
+                    for accession in pdb_accessions:
+                        pdbl.retrieve_pdb_file(
+                            f"{accession}",
+                            file_format=args.pdb,
+                            pdir=args.output,
+                        )
+                else:
+                    for accession in pdb_accessions:
+                        pdbl.retrieve_pdb_file(
+                            f"{accession}",
+                            file_format=args.pdb,
+                            pdir=os.getcwd,
+                        )
+
+    return
+
+
+def get_genbank_accession(df_row, df_name, row_index, logger):
+    """Retrieve GenBank accession for protein in the dataframe (df) row.
+
+    Retrieve the first GenBank accession becuase if multiple are given, CAZy only links
+    to the first GenBank accession.
+
+    :param df_row: Pandas series, from protein protein dataframe
+    :param df_name: str, name of the Pandas dataframe from which row was retrieved
+    :param row_index: int, index of the row in the protein dataframe
+    :param logger: logger object
+
+    Return two strings, GenBank accession of the protein and the protein's CAZy family.
+    """
+    family = df_row[1]
+    genbank_cell = df_row[4]
+
+    if type(genbank_cell) is float:
+        return None, None
+
+    # Retrieve the first accession number, and separate from assoicated HTML link
+    first_accession = genbank_cell.split(" ")[0]
+
+    # check result is in expected genbank format
+    try:
+        re.match(r"\D{3}\d+.\d+", first_accession).group()
+    except AttributeError:
+        logger.warning(
+            f"Could not return accession for protein in row {row_index} in\n"
+            f"{df_name}.\n"
+            "Not retrieving FASTA file for this protein."
+        )
+        return None, None
+
+    return first_accession, family
+
+
+def get_pdb_accessions(df_row, df_name, logger):
+    """Retrieve PDB accession for protein in the dataframe (df) row.
+
+    Retrieves all PDB accessions becuase multiple are accession can be given. Duplicate accession
+    are not retrieved.
+
+    :param df_row: Pandas series, from protein protein dataframe
+    :param df_name: str, name of the Pandas dataframe from which row was retrieved
+    :param row_index: int, index of the row in the protein dataframe
+    :param logger: logger object
+
+    Return list of PDB accessions.
+    """
+    pdb_cell = df_row[6]
+
+    # Check if null value is stored in pdb cell
+    if type(pdb_cell) is float:
+        return None
+
+    # Separate the PDB accession
+    pdb_accessions = pdb_cell.split(",\n")
+
+    index = 0
+    for index in range(len(pdb_accessions)):
+        # remove html links
+        pdb_accessions[index] = pdb_accessions[index].split(" ")[0]
+        # remove additional data in square brackest
+        pdb_accessions[index] = pdb_accessions[index].split("[")[0]
+
+    # remove duplicate accessions
+    pdb_accessions = list(dict.fromkeys(pdb_accessions))
+
+    return pdb_accessions
+
+
+def get_pdb_structures(protein_dataframe, df_name, args, logger):
+    """Coordinate the retrieval of CAZyme structures from PDB.
+
+    :param protein_dataframe: Pandas dataframe, dataframe containing protein data from CAZy
+    :param df_name: str, name of the CAZy dataframe
+    :param args: cmd args parser
+    :param logger: logger object
+
+    Return nothing.
+    """
+    # build PDBList object
+    pdbl = PDBList()
+    index = 0
+    for index in tqdm(
+        range(len(protein_dataframe["Protein_name"])),
+        desc=f"Downloading {df_name} structures",
+    ):
+        # Retrieve accession from GenBank cell
+        df_row = protein_dataframe.iloc[index]
+        pdb_accessions = get_pdb_accessions(df_row, df_name, logger)
+
+        if pdb_accessions is None:
+            continue
+
+        # Get structures from RSCB
+        if args.pdb_output is not None:
+            for accession in pdb_accessions:
+                pdbl.retrieve_pdb_file(
+                    f"{accession}",
+                    file_format=args.pdb,
+                    pdir=args.pdb_output,
+                )
+
+        else:
+            if args.output is not sys.stdout:
+                for accession in pdb_accessions:
+                    pdbl.retrieve_pdb_file(
+                        f"{accession}",
+                        file_format=args.pdb,
+                        pdir=args.output,
+                    )
+            else:
+                for accession in pdb_accessions:
+                    pdbl.retrieve_pdb_file(
+                        f"{accession}",
+                        file_format=args.pdb,
+                        pdir=os.getcwd,
+                    )
+
+    return
+
+
+def get_genbank_fasta(dataframe, df_name, args, logger):
+    """Retrieve GenBank accessions, coordinate downloading associated GenBank FASTA file.
+
+    :param df_row: Pandas series, from protein protein dataframe
+    :param df_name: str, name of the Pandas dataframe from which row was retrieved
+    :param row_index: int, index of the row in the protein dataframe
+    :param logger: logger object0345 454 111
+
+    Return  nothing.
+    """
+    # set user email address for Entrez
+    Entrez.email = args.genbank
+
+    index = 0
+    for index in tqdm(
+        range(len(dataframe["Protein_name"])), desc=f"Downloading {df_name} GenBank FASTAs"
+    ):
+        # Retrieve accession from GenBank cell
+        df_row = dataframe.iloc[index]
+        accession, cazy_family = get_genbank_accession(df_row, df_name, index, logger)
+
+        if accession is None:
+            continue
+
+        # create file name
+        file_name = f"{accession}_{cazy_family}.fasta"
+        if args.genbank_output is not sys.stdout:
+            file_name = args.genbank_output / file_name
+
+        download_fasta(accession, file_name, args, logger)
+
+    return
+
+
+def download_fasta(accession, file_name, args, logger):
+    """Download FASTA file from GenBank FTP server.
+
+    :param accession: str, accession of protein
+    :param file_name: str, path to output file desintation
+    :param args: cmd args parser
+    :param logger: logger object
+
+    Returning nothing.
+    """
+    if args.genbank_output is not sys.stdout:
+        if file_name.exists():
+            logger.warning(f"FASTA file {file_name} aleady exists, not downloading again")
+            return
+
+    handle = entrez_retry(
+        logger,
+        Entrez.efetch,
+        db="protein",
+        id=accession,
+        rettype="fasta",
+        retmode="text",
+    )
+
+    if handle is None:
+        logger.warning(f"Failed to download FASTA file for {accession}")
+        return
+
+    for record in SeqIO.parse(handle, "fasta"):
+        if args.genbank_output is not sys.stdout:
+            with open(file_name, "w") as fh:
+                SeqIO.write(record, fh, "fasta")
+        else:
+            SeqIO.write(record, sys.stdout, "fasta")
+
+    return
+
+
+def entrez_retry(logger, entrez_func, *func_args, **func_kwargs):
+    """Call to NCBI using Entrez.
+    Maximum number of retries is 10, retry initated when network error encountered.
+    :param logger: logger object
+    :param retries: parser argument, maximum number of retries excepted if network error encountered
+    :param entrez_func: function, call method to NCBI
+    :param *func_args: tuple, arguments passed to Entrez function
+    :param ** func_kwargs: dictionary, keyword arguments passed to Entrez function
+    Returns record.
+    """
+    record, retries, tries = None, 10, 0
+
+    while record is None and tries < retries:
+        try:
+            record = entrez_func(*func_args, **func_kwargs)
+
+        except IOError:
+            # log retry attempt
+            if tries < retries:
+                logger.warning(
+                    f"Network error encountered during try no.{tries}.\nRetrying in 10s",
+                    exc_info=1,
+                )
+                time.sleep(10)
+            tries += 1
+
+    if record is None:
+        logger.error(
+            "Network error encountered too many times. Exiting attempt to call to NCBI"
+        )
+        return
+
+    return record
diff --git a/cazy_webscraper/scraper_config.yaml b/cazy_webscraper/scraper_config.yaml
new file mode 100644
index 00000000..f3cd6f74
--- /dev/null
+++ b/cazy_webscraper/scraper_config.yaml
@@ -0,0 +1,15 @@
+# Under 'classes' list class from which all proteins will retrieved
+# Under each families respective name, list the specific families/subfamilies to be scraped
+# Write the FULL family name, e.g. 'GH1', NOT only its number, e.g. '1'
+# To list multiple families, each familiy must be on a new line starting indented once
+# relative to the parent class name, and the name written within quotation marks.
+# For more information on writing lists in Yaml please see:
+# https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html 
+classes:  # classes from which all proteins will be retrieved
+Glycoside Hydrolases (GHs):
+  - "GH4"
+GlycosylTransferases (GTs):
+Polysaccharide Lyases (PLs):
+Carbohydrate Esterases (CEs):
+Auxiliary Activities (AAs):
+Carbohydrate-Binding Modules (CBMs):
\ No newline at end of file
diff --git a/cazy_webscraper/utilities/__init__.py b/cazy_webscraper/utilities/__init__.py
new file mode 100644
index 00000000..45c5e062
--- /dev/null
+++ b/cazy_webscraper/utilities/__init__.py
@@ -0,0 +1,207 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Author:
+# Emma E. M. Hobbs
+
+# Contact
+# eemh1@st-andrews.ac.uk
+
+# Emma E. M. Hobbs,
+# Biomolecular Sciences Building,
+# University of St Andrews,
+# North Haugh Campus,
+# St Andrews,
+# KY16 9ST
+# Scotland,
+# UK
+
+# The MIT License
+"""Build cmd-line parser and logger."""
+
+import argparse
+import logging
+import sys
+
+from pathlib import Path
+from typing import List, Optional
+
+
+def build_parser(argv: Optional[List] = None):
+    """Return ArgumentParser parser for script."""
+    # Create parser object
+    parser = argparse.ArgumentParser(
+        prog="cazy_webscraper.py",
+        description="Scrapes the CAZy database",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+
+    # Add optional arguments to parser
+
+    # Add option to specify path to configuration file
+    parser.add_argument(
+        "-c",
+        "--config",
+        type=Path,
+        metavar="config file",
+        default=None,
+        help="Path to configuration file. Default: None, scrapes entire database",
+    )
+
+    # Add option on how to split data
+    parser.add_argument(
+        "-d",
+        "--data_split",
+        choices=[None, "class", "family"],
+        type=str,
+        default=None,
+        help="How data is to be split. Not split=all, split by class=class, split by family=family",
+    )
+
+    # Add option to force file over writting
+    parser.add_argument(
+        "-f",
+        "--force",
+        dest="force",
+        action="store_true",
+        default=False,
+        help="Force file over writting",
+    )
+
+    # Add option to download FASTA file for protein from GenBank
+    parser.add_argument(
+        "-g",
+        "--genbank",
+        type=str,
+        metavar="Email address of user",
+        default=None,
+        help="Enable FASTA files from GenBank, and user email required for Entrez",
+    )
+
+    # Add log file name option
+    # If not given, no log file will be written out
+    parser.add_argument(
+        "-l",
+        "--log",
+        type=Path,
+        metavar="log file name",
+        default=None,
+        help="Defines log file name and/or path",
+    )
+
+    # Add option to prevent over writing of existing files
+    # and cause addition of files to output directory
+    parser.add_argument(
+        "-n",
+        "--nodelete",
+        dest="nodelete",
+        action="store_true",
+        default=False,
+        help="enable/disable deletion of exisiting files",
+    )
+
+    # Add option to specify output directory to write output dataframes to
+    parser.add_argument(
+        "-o",
+        "--output",
+        type=Path,
+        metavar="output file name",
+        default=sys.stdout,
+        help="Output filename",
+    )
+
+    # Add option to specify ouput directory for writing out fasta files from GenBank to
+    parser.add_argument(
+        "-genbank_output",
+        type=Path,
+        metavar="output file name",
+        default=sys.stdout,
+        help="Output filename",
+    )
+
+    # Add option to specift output directory for writing out PDB structure files to
+    parser.add_argument(
+        "-pdb_output",
+        type=Path,
+        metavar="output file name",
+        default=None,
+        help="Output filename",
+    )
+
+    # Add option to download FASTA file for protein from GenBank
+    parser.add_argument(
+        "-p",
+        "--pdb",
+        choices=[None, "mmCif", "pdb", "xml", "mmtf", "bundle"],
+        type=str,
+        default=None,
+        help="Enable downloading of protein structures in XXXX format from PDB",
+    )
+
+    # Add option to enable number of times to retry scraping
+    parser.add_argument(
+        "-r",
+        "--retries",
+        type=int,
+        default=0,
+        help="Number of times to retry scraping a family or class page if error encountered",
+    )
+
+    # Add option to enable retrieval of subfamilies
+    parser.add_argument(
+        "-s",
+        "--subfamilies",
+        dest="subfamilies",
+        action="store_true",
+        default=False,
+        help="Enable retrieval of subfamilies from CAZy",
+    )
+
+    # Add option for more detail (verbose) logging
+    parser.add_argument(
+        "-v",
+        "--verbose",
+        dest="verbose",
+        action="store_true",
+        default=False,
+        help="Set logger level to 'INFO'",
+    )
+
+    if argv is None:
+        # parse command-line
+        return parser
+    else:
+        # return namespace
+        return parser.parse_args(argv)
+
+
+def build_logger(script_name, args) -> logging.Logger:
+    """Return a logger for this script.
+    Enables logger for script, sets parameters and creates new file to store log.
+    :param script_name: str, name of script
+    :param args: parser argument
+    Return logger object.
+    """
+    logger = logging.getLogger(script_name)
+
+    # Set format of loglines
+    log_formatter = logging.Formatter(
+        script_name + ": {} - {}".format("%(asctime)s", "%(message)s")
+    )
+
+    if args.verbose is True:
+        logger.setLevel(logging.INFO)
+    else:
+        logger.setLevel(logging.WARNING)
+
+    # Setup console handler to log to terminal
+    console_log_handler = logging.StreamHandler()
+    console_log_handler.setFormatter(log_formatter)
+    logger.addHandler(console_log_handler)
+
+    # Setup file handler to log to a file
+    if args.log is not None:
+        file_log_handler = logging.FileHandler(args.log)
+        file_log_handler.setFormatter(log_formatter)
+        logger.addHandler(file_log_handler)
+
+    return logger
diff --git a/requirements-dev.txt b/requirements-dev.txt
new file mode 100644
index 00000000..7363d926
--- /dev/null
+++ b/requirements-dev.txt
@@ -0,0 +1,14 @@
+autopep8
+bandit
+black
+coverage
+flake8
+lxml
+mypy
+pydocstyle
+pycodestyle
+pylint
+pytest
+pytest-cov
+pytest-ordering
+sphinx
diff --git a/requirements-pip.txt b/requirements-pip.txt
new file mode 100644
index 00000000..4170c03e
--- /dev/null
+++ b/requirements-pip.txt
@@ -0,0 +1,1 @@
+sphinx-rtd-theme
\ No newline at end of file
diff --git a/scraper/cazy_webscraper.py b/scraper/cazy_webscraper.py
index dba1d718..d9107413 100644
--- a/scraper/cazy_webscraper.py
+++ b/scraper/cazy_webscraper.py
@@ -29,13 +29,7 @@
 :cmd_args --verbose: change logger level from warning to info, verbose logging
 
 :func main: coordinate scraping of CAZy database
-:func get_cazy_class_urls: retrieve URLs to CAZy class summary pages from CAZy homepage
-:func get_cazy_family_urls: retrieve URLs to families on CAZy class summary page
-:func get_subfamily_links: retrieve URLs to subfamilies on CAZy class summary page
-:func parse_family: build Family class object to represent CAZy family
-:func parse_family_pages: retrieve all URLs to pages containing protein records for CAZy family
-:func parse_proteins: retrieve protein records from protein table page
-:func row_to_protein: parse the protein record to build a Protein class object
+:func get_class_urls: retrieve URLs to CAZy class summary pages from CAZy homepage
 :func browser_decorator: decorator for get_page() to coordinate retrying failed connections
 :func get_page: connect to webpage and retrieval page as BeautifulSoup4 object
 
@@ -46,112 +40,13 @@
 import logging
 import re
 import sys
-import time
 
-import numpy as np
-
-from collections import defaultdict
+from datetime import datetime
 from typing import List, Optional
-from requests.exceptions import ConnectionError, MissingSchema
-from urllib3.exceptions import HTTPError, RequestError
-
-import mechanicalsoup
 
 from tqdm import tqdm
 
-from scraper import utilities, file_io, parse
-
-
-class Protein:
-    """A single protein.
-
-    Each protein has a name, source organism (source), and links to external databases. The links to
-    external databases are stored in a dictionary, keyed by the external database name ('str') with
-    'list' values becuase there may be multiple links per database.
-
-    Multiple 'synonym' GenBank accession numbers maybe listed for a single protein. CAZy only
-    hyperlinks the first listed accession number. This accession is the one listed for the protein,
-    because is presumed to be the accession used by CAZy in their classification. All other listed
-    GenBank accessions are regarded as synonyms, including for example splice variants and identical
-    protein sequence submissions.
-    """
-
-    def __init__(self, name, family, ec, source, links=None, genbank_synonyms=None):
-        self.name = name
-        self.family = family
-        self.ec = ec
-        self.source = source
-        if links is None:
-            self.links = defaultdict(list)
-        else:
-            self.links = links
-        self.genbank_synonyms = genbank_synonyms
-
-    def __str__(self):
-        """Create representative string of class object"""
-        return f"{self.name} ({self.family} {self.source}): links to {self.links.keys()}"
-
-    def __repr__(self):
-        """Create representative object"""
-        return(
-            (
-                f"<Protein: {id(self)}: {self.name}, {self.family} "
-                f"({self.source}), {len(self.links)} to external databases>"
-            )
-        )
-
-    def get_protein_dict(self):
-        """Return a dictionary containing all the data of the protein."""
-        protein_dict = {"Protein_name": [self.name], "CAZy_family": [self.family]}
-
-        if len(self.ec) == 0:
-            protein_dict["EC#"] = [np.nan]
-        elif len(self.ec) == 1:
-            protein_dict["EC#"] = self.ec
-        else:
-            ec_string = "\n".join(self.ec)
-            protein_dict["EC#"] = [ec_string]
-
-        protein_dict["Source_organism"] = [self.source]
-
-        if type(self.links) is dict:
-            for database in ["GenBank", "UniProt", "PDB/3D"]:
-                try:
-                    if len(self.links[database]) == 1:
-                        protein_dict[database] = self.links[database]
-                    else:
-                        accession_string = ",\n".join(self.links[database])
-                        protein_dict[database] = [accession_string]
-                except KeyError:
-                    protein_dict[database] = [np.nan]
-        else:
-            for database in ["GenBank", "UniProt", "PDB/3D"]:
-                protein_dict[database] = [np.nan]
-        return protein_dict
-
-
-class Family:
-    """A single CAZy family."""
-
-    members = set()  # holds Protein instances
-
-    def __init__(self, name, cazy_class):
-        self.name = name
-        self.cazy_class = cazy_class
-
-    def __str__(self):
-        return f"CAZy family {self.name}: {len(self.members)} protein members"
-
-    def __repr__(self):
-        return f"<Family: {id(self)}: {self.name}, {len(self.members)} protein members"
-
-    def get_proteins(self):
-        """Return a list of all protein members of the CAZy family."""
-        return self.members
-
-    def get_family_name(self):
-        """Return family name"""
-        return self.name
+from scraper import crawler, file_io, parse, utilities
 
 
 def main(argv: Optional[List[str]] = None, logger: Optional[logging.Logger] = None):
@@ -163,6 +58,8 @@ def main(argv: Optional[List[str]] = None, logger: Optional[logging.Logger] = No
     families specified to be scraped in the configration file.
     """
     # Program preparation
+    time_stamp = datetime.now().strftime("%Y-%m-%d--%H-%M-%S")  # used in naming files
+
     if argv is None:
         parser = utilities.build_parser()
         args = parser.parse_args()
@@ -176,7 +73,7 @@ def main(argv: Optional[List[str]] = None, logger: Optional[logging.Logger] = No
     if args.output is not sys.stdout:
         file_io.make_output_directory(args.output, logger, args.force, args.nodelete)
 
-    if (args.genbank is not None):
+    if args.genbank is not None:
         # create directory to write FASTA files to
         if (args.genbank_output is not sys.stdout) and (args.genbank_output != args.output):
             file_io.make_output_directory(args.genbank_output, logger, args.force, args.nodelete)
@@ -189,6 +86,8 @@ def main(argv: Optional[List[str]] = None, logger: Optional[logging.Logger] = No
     if args.subfamilies is True:
         logger.warning("Enabled to retrieve subfamilies")
 
+    max_tries = args.retries + 1  # maximum number of times to try scraping a CAZy
+
     # retrieve configuration data
     file_io_path = file_io.__file__
     excluded_classes, config_dict, cazy_dict = file_io.parse_configuration(
@@ -204,7 +103,16 @@ def main(argv: Optional[List[str]] = None, logger: Optional[logging.Logger] = No
     cazy_home = "http://www.cazy.org"  # the CAZy homepage URL
 
     # Retrieve data from CAZy database
-    get_cazy_data(cazy_home, excluded_classes, config_dict, cazy_dict, logger, args)
+    get_cazy_data(
+        cazy_home,
+        excluded_classes,
+        config_dict,
+        cazy_dict,
+        max_tries,
+        time_stamp,
+        logger,
+        args,
+    )
 
     logger.info(
         (
@@ -215,7 +123,16 @@ def main(argv: Optional[List[str]] = None, logger: Optional[logging.Logger] = No
     )
 
 
-def get_cazy_data(cazy_home, excluded_classes, config_dict, cazy_dict, logger, args):
+def get_cazy_data(
+    cazy_home,
+    excluded_classes,
+    config_dict,
+    cazy_dict,
+    max_tries,
+    time_stamp,
+    logger,
+    args,
+):
     """Coordinate retrieval of data from the CAZy website.
 
     This function coordinates the crawling through the CAZy website by calling the appropriate
@@ -225,79 +142,105 @@ def get_cazy_data(cazy_home, excluded_classes, config_dict, cazy_dict, logger, a
     :param excluded_classes: list, list of classes to not scrape from CAZy
     :param config_dict: dict, user defined configuration of the scraper
     :param cazy_dict: dict, dictionary of excepct CAZy synonyms for CAZy classes
+    :param max_tries: int, maximum number of times to scrape CAZy if errors are encountered
+    :param time_stamp: str, data and time scrape was initiated
     :param logger: logger object
     :param args: cmd args parser
 
     Return nothing.
     """
+    # List of urls that were failed to be scraped
+    failed_url_scrapes = []
 
     # retrieve links to CAZy class pages
-    class_pages = get_cazy_class_urls(cazy_home, excluded_classes, logger)
-
-    try:
-        if len(class_pages) == 0:
-            logger.error(
-                "Failed to retrieval URLs to CAZy class pages\n"
-                "Terminating program"
-                )
-            sys.exit(1)
-    except TypeError:  # rased when class_pages is None
-        logger.error(
-            "Failed to retrieval URLs to CAZy class pages\n"
-            "Terminating program"
-            )
-        sys.exit(1)
+    # nested lists of [class_url, number_of_attempted_class_url_scrapes]
+    class_urls = get_class_urls(cazy_home, excluded_classes, max_tries, logger)
 
     all_data = []  # stores all Family class objects if not splitting the data
 
     logger.info("Starting retrieval of CAZy families")
 
     # scrape each retrieved class page
-    for class_url in tqdm(class_pages, desc="Parsing CAZy classes"):
+    for class_url in tqdm(class_urls, desc="Parsing CAZy classes"):
 
         # retrieve class name from url
-        class_name = class_url[20: -5]
+        class_name = class_url[0][20:-5]
         #  convert retrieved class name to synonym used in configuration file
         for key in cazy_dict:
             if class_name in cazy_dict[key]:
                 class_name = key
 
         # retrieve URLs to families under current working CAZy class
-        family_urls = get_cazy_family_urls(class_url, cazy_home, class_name, args, logger)
+        # nested list of [family_url, number_of_attempts_to_scrape_family_all_page]
+        family_urls = crawler.get_cazy_family_urls(
+            class_url[0],
+            cazy_home,
+            class_name,
+            args,
+            logger,
+        )
 
         if family_urls is None:  # couldn't retrieve URLs to family for working CAZy class
-            continue  # got to next CAZy class, issue logged in get_cazy_family_urls()
+            # add one to the number of scrapping attempts
+            class_url[1] += 1
+
+            if class_url[1] == max_tries:  # max number of attempts to scrape has been met
+                failed_url_scrapes += (
+                    f"{class_url[0]} - no CAZy familes from this class were scraped"
+                )
+                continue
+            else:
+                class_urls += class_url  # retry scraping Class page after the other Class URLs
+                continue
 
         families = []  # store Family class objects if splitting data be class
+
         logger.info("Starting retrieval of protein records of protein records from families")
 
-        # Scrape the familes for the current working CAZy class, retrieving protein data
+        # Scrape the familes of the current CAZy class, retrieving protein data
+
         if (config_dict is None) or (config_dict[class_name] is None):
             # No (sub)families were specified, therefore, scraping all families of the CAZy class
 
             for family_url in tqdm(family_urls, desc="Parsing CAZy families"):
                 # check url format is correct
                 try:
-                    re.match(rf"{cazy_home}/(\D\D|\D\D\D)(\d+|\d_\d+).html", family_url).group()
+                    re.match(
+                        r"http://www.cazy.org/(\D{2,3})(\d+|\d+_\d+).html", family_url[0]
+                    ).group()
                 except AttributeError:
                     logger.warning(
-                        (
-                            f"Formate of URL {family_url} is incorrect.\n"
-                            "Will not attempt to scrape this URL."
-                        )
+                        f"Formate of URL {family_url[0]} is incorrect.\n"
+                        "Will not attempt to scrape this URL."
                     )
+                    failed_url_scrapes += f"{family_url[0]} - url format was incorrect"
                     continue
 
                 family = None
-                family_name = family_url[(len(cazy_home) + 1): -5]
+                family_name = family_url[0][(len(cazy_home) + 1): -5]
                 # build family object, populated by Proteins catalogued under the CAZy family
-                family = parse_family(family_url, family_name, cazy_home, logger)
+                family = crawler.parse_family(family_url[0], family_name, cazy_home, logger)
+
+                # [family_object, error]
+                if family[1] is not None:  # Scraping family '_all' page was unsuccessful
+                    # add one to the number of attempted scrapes the CAZy family's '_all' page
+                    family_url[1] += 1
 
+                    if family_url[1] == max_tries:  # max number of scraping attempts reached
+                        failed_url_scrapes += (
+                            f"{family_url[0]} - the following error was raised {family[2]}"
+                        )
+                        continue
+                    else:
+                        family_urls += family_url
+                        continue
+
+                # store the family if scraped successfully
                 if args.data_split == "family":
                     logger.info(f"Data split by Family. Writing out df for {family_name}")
-                    parse.proteins_to_dataframe([family], args, logger)
+                    parse.proteins_to_dataframe([family[0]], time_stamp, args, logger)
                 else:
-                    families.append(family)
+                    families.append(family[0])
 
         else:
             # scrape only (sub)families specified in the config file
@@ -305,40 +248,57 @@ def get_cazy_data(cazy_home, excluded_classes, config_dict, cazy_dict, logger, a
             for family_url in tqdm(family_urls, desc="Parsing CAZy families"):
                 # check url format is correct
                 try:
-                    re.match(rf"{cazy_home}/(\D\D|\D\D\D)(\d+|\d_\d+).html", family_url).group()
+                    re.match(
+                        r"http://www.cazy.org/(\D{2,3})(\d+|\d+_\d+).html", family_url[0]
+                    ).group()
                 except AttributeError:
                     logger.warning(
                         (
-                            f"Formate of URL {family_url} is incorrect.\n"
+                            f"Formate of URL {family_url[0]} is incorrect.\n"
                             "Will not attempt to scrape this URL."
                         )
                     )
                     continue
 
                 family = None
-                family_name = family_url[(len(cazy_home) + 1): -5]
+                family_name = family_url[0][(len(cazy_home) + 1) : -5]
 
                 # Allows retrieval of subfamilies when only the parent CAZy family was named in the
                 # config file
                 if (args.subfamilies is True) and (family_name.find("_") != -1):
-                    name_check = family_name[:(family_name.find("_"))]
+                    name_check = family_name[: (family_name.find("_"))]
                 else:
                     name_check = family_name
 
                 if name_check in config_dict[class_name]:
-                    # build family object populated with Proteins catalogued under the CAZy family
-                    family = parse_family(family_url, family_name, cazy_home, logger)
-
+                    # build family object, populated by Proteins catalogued under the CAZy family
+                    family = crawler.parse_family(family_url[0], family_name, cazy_home, logger)
+
+                    # [family_object, error]
+                    if family[1] is not None:  # Scraping family '_all' page was unsuccessful
+                        # add one to the number of attempted scrapes the CAZy family's '_all' page
+                        family_url[1] += 1
+
+                        if family_url[1] == max_tries:  # max number of scraping attempts reached
+                            failed_url_scrapes += (
+                                f"{family_url[0]} - the following error was raised {family[2]}"
+                            )
+                            continue
+                        else:
+                            family_urls += family_url
+                            continue
+
+                    # store the family if scraped successfully
                     if args.data_split == "family":
                         logger.info(f"Data split by Family. Writing out df for {family_name}")
-                        parse.proteins_to_dataframe([family], args, logger)
+                        parse.proteins_to_dataframe([family[0]], time_stamp, args, logger)
                     else:
-                        families.append(family)
+                        families.append(family[0])
 
-        if (args.data_split == "class"):
+        if args.data_split == "class":
             if len(families) != 0:
                 logger.info(f"Data split by Class. Writing out df for {class_name}")
-                parse.proteins_to_dataframe(families, args, logger)
+                parse.proteins_to_dataframe(families, time_stamp, args, logger)
             else:
                 logger.warning(f"Didn't retrieve any families for {class_name}")
 
@@ -348,379 +308,45 @@ def get_cazy_data(cazy_home, excluded_classes, config_dict, cazy_dict, logger, a
     if args.data_split is None:
         if len(all_data) != 0:
             logger.info("Data was not split. Writing all retrieved data to a single df")
-            parse.proteins_to_dataframe(all_data, args, logger)
+            parse.proteins_to_dataframe(all_data, time_stamp, args, logger)
         else:
             logger.warning("Didn't retrieve any protein data from CAZy")
 
+    # write out URLs which failed to be scaped
+    file_io.write_out_failed_scrapes(failed_url_scrapes, time_stamp, args, logger)
+
     return
 
 
-def get_cazy_class_urls(cazy_home, excluded_classes, logger):
-    """Returns a list of CAZy class main/home page URLs for each specified class as the CAZy site.
+def get_class_urls(cazy_home, excluded_classes, max_tries, logger):
+    """Retrieve class urls, add storage of number of attempted scrapes of each class URL.
 
     :param cazy_url: str, URL to the CAZy home page.
     :param excluded_classes: list, list of CAZy classes not to be scraped
+    :param max_tries: int, maximum number of times to try scrape if errors are encountered
     :param logger: logger object
 
-    Return list of URLs.
+    Return list of CAZy class URLs. Each item is a list of [URL, 0]
+    - 0 is used to count number of attempted connections.
     """
-    logger.info("Retrieving URLs to summary CAZy class pages")
 
-    # define items to be excluded from returned class list, ALWAYS exlide links to genomes
-    if excluded_classes is not None:
-        exclusions = tuple(["<strong>Genomes</strong>"] + excluded_classes)
-    else:
-        exclusions = ("<strong>Genomes</strong>")
-
-    # scrape the home page
-    home_page = get_page(cazy_home)
-    if home_page[0] is None:
-        logger.error(
-            (
-                "Failed to connect to CAZy home-page after 10 attempts.\n"
-                "The following error was raised:\n"
-                f"{home_page[1]}"
-                "Could not retrieve URLs to CAZy classes.\n"
-                "Check the network connection.\nTerminating program."
-            )
-        )
-        sys.exit(1)
+    class_urls = crawler.get_cazy_class_urls(cazy_home, excluded_classes, max_tries, logger)
 
     try:
-        return [f"{cazy_home}/{_['href']}" for _ in
-                home_page[0].find_all("a", {"class": "spip_out"}) if
-                (not _["href"].startswith("http")) and
-                (str(_.contents[0]) not in exclusions)]
-    except AttributeError:  # raise if can't find results with find_all("a", {"class": "spip_out"})
-        return None
-
-
-def get_cazy_family_urls(class_url, cazy_home, class_name, args, logger):
-    """Retrieve all protein members of each CAZy family within the given CAZy class.
-
-    :param class_url: str, URL to the CAZy class
-    :param cazy_home: str, URL to CAZy home page
-    :param class_name: str, name of CAZy class
-    :param args: args parser object
-    :param logger: logger object
-
-    Returns list of URLs to family pages.
-    """
-    logger.info(f"Retrieving URLs to families under {class_name}")
-
-    # scrape the class page
-    class_page = get_page(class_url)
-
-    if class_page[0] is None:
-        logger.error(
-            (
-                f"Failed to connect to {class_url} after 10 attempts.\n"
-                "The following error was raised:\n"
-                f"{class_page[1]}"
-                "Could not retrieve URLs to CAZy famileis for this class.\n"
-                "This class will be skipped during the scraping process."
-            )
-        )
-        return None
-
-    # retrieve the <h3> element that titles the div section containing the tables of family links
-    family_h3_element = [_ for _ in class_page[0].find_all("h3", {"class": "spip"}) if str(_.contents[0]).strip() == "Tables for Direct Access"][0]
-
-    # retrieve all tables within the parent div section of the <h3> element
-    tables = family_h3_element.parent.find_all("table")
-
-    # tables[0] is the table containing links to CAZy families
-    # tables[1] is the table containing the link to unclassified proteins
-
-    family_urls = family_urls = [f"{cazy_home}/{_['href']}" for _ in tables[0].find_all("a")]
-    family_urls.append(f"{cazy_home}/{tables[1].a['href']}")
-    if args.subfamilies is True:
-        subfam_urls = get_subfamily_links(family_h3_element, cazy_home, logger)
-        if subfam_urls is not None:
-            family_urls += subfam_urls
-
-    return family_urls
-
-
-def get_subfamily_links(family_h3_element, cazy_home, logger):
-    """Retrieve URL links to CAZy subfamilies.
-
-    :param family_h3_element: bs4.element.Tag, h3 element titling the page div
-    :param cazy_home: str, URL to CAZy home_page
-    :param logger: logger object
-
-    Return list of URLs to subfamilies.
-    """
-    parent_div = family_h3_element.parent
-    all_links = parent_div.find_all("a")
-
-    pattern = re.compile(r"\D+?\d+?_\d+?\.html")
-
-    urls = []  # empty list to store subfamily URLs
-
-    for link in all_links:
-        try:
-            search_result = re.search(pattern, link["href"])
-            urls.append(f"{cazy_home}/{search_result.group()}")
-        except (KeyError, AttributeError) as error:
-            # KeyError raised if link does not have ['href']
-            # AttributeError error raised if search_result is None becuase not subfam link
-            pass
-
-    if len(urls) == 0:
-        return
-    else:
-        return urls
-
-
-def parse_family(family_url, family_name, cazy_home, logger):
-    """Returns a Family object with Protein members, scraped from CAZy.
-
-    :param family_url: str, URL to CAZy family summary page
-    :param family_name: str, name of CAZy family
-    :param cazy_home: str, URL to CAZy home page
-    :param logger: logger object
-
-    Return Family object.
-    """
-    logger.info(f"Starting retrieval of proteins for {family_name}")
-
-    # retrieve class from family name
-    pattern = re.compile(r"\D+")
-    search_result = re.match(pattern, family_name)
-    cazy_class = search_result.group()
-
-    family = Family(family_name, cazy_class)
-    family.members = set()
-
-    for protein in tqdm(
-        parse_family_pages(family_url, family_name, cazy_home, logger),
-        desc=f"Retrieving proteins from {family_name}"
-    ):
-        family.members.add(protein)
-
-    return family
-
-
-def parse_family_pages(family_url, family_name, cazy_home, logger):
-    """Retrieve all protein records for given CAZy family.
-    Protein records are listed in a pagination method, with 1000 proteins per page.
-
-    :param family_url: str, URL to CAZy family main page
-    :param family_name: str, name of CAZy family
-    :param cazy_home: str, URL to CAZy home page
-    :param logger: logger object
-
-    Return list of protein records.
-    """
-    logger.info(f"Retrieving URLs to all pages containing proteins for {family_url}")
-
-    # compile URL to first family page of protein records
-    first_pagination_url = family_url.replace(".html", "_all.html")
-
-    # check url formating
-    try:
-        re.match(rf"{cazy_home}/(\D\D|\D\D\D)(\d+|\d_\d+)_all.html", first_pagination_url).group()
-    except AttributeError:
-        logger.warning(
-            f"Incorrect formating of first protein table page URL: {first_pagination_url}\n"
-            "Will not try and connect to this URL."
-        )
-        return []
-
-    first_pagination_page = get_page(first_pagination_url)
-
-    if first_pagination_page[0] is None:
-        logger.warning(
-            (
-                f"Could not connect to {family_url}\n"
-                "The following error was raised:\n"
-                f"{first_pagination_page[1]}"
-                f"No protein records for CAZy family {family_name} will be retried."
-            )
-        )
-        return []
-
-    protein_page_urls = [first_pagination_url]
-
-    # retrieve the URL to the final page of protein records in the pagination listing
-    try:
-        last_pagination_url = first_pagination_page[0].find_all(
-            "a", {"class": "lien_pagination", "rel": "nofollow"})[-1]
-    except IndexError:  # there is no pagination; a single-query entry
-        last_pagination_url = None
-
-    if last_pagination_url is not None:
-        url_prefix = last_pagination_url["href"].split("PRINC=")[0] + "PRINC="
-        last_princ_no = int(last_pagination_url["href"].split("PRINC=")[-1].split("#pagination")[0])
-        url_suffix = "#pagination" + last_pagination_url["href"].split("#pagination")[-1]
-
-        # Build list of urls to all pages in the pagination listing, increasing the PRINC increment
-        protein_page_urls.extend([f"{cazy_home}/{url_prefix}{_}{url_suffix}" for _ in
-                                  range(1000, last_princ_no + 1000, 1000)])
-
-    # Process all URLs into a single collection - a generator
-    # return (y for x in (parse_proteins(url, family_name, logger)
-    # for url in protein_page_urls) for y in x)
-
-    return (
-        y for x in (
-            parse_proteins(url, family_name, logger) for url in protein_page_urls
-        ) for y in x
-    )
-
-
-def parse_proteins(protein_page_url, family_name, logger):
-    """Returns generator of Protein objects for all protein rows on a single CAZy family page.
-
-    :param protein_page_url, str, URL to the CAZy family page containing protein records
-    :param family_name: str, name of CAZy family
-    :param logger: logger object
-
-    Return generator object.
-    """
-    logger.info(f"Retrieving proteins from {protein_page_url}")
-
-    protein_page = get_page(protein_page_url)
-
-    if protein_page[0] is None:
-        logger.warning(
-            (
-                f"Could not connect to {protein_page_url}\n"
-                "The following error was raised:\n"
-                f"{protein_page[1]}\n"
-                f"No protein records from this page will be retried."
-            )
-        )
-        return
-
-    # retrieve protein record table
-    protein_table = protein_page[0].find_all("table", {"class": "listing"})[0]
-    protein_rows = [_ for _ in protein_table.descendants if (_.name == "tr") and
-                    ("id" not in _.attrs) and ("class" not in _.attrs)]
-
-    for row in protein_rows:
-        yield row_to_protein(row, family_name)
-
-
-def row_to_protein(row, family_name):
-    """Returns a Protein object representing a single protein row from a CAZy family protein page.
-
-    Each row, in order, contains the protein name, EC number, source organism, GenBank ID(s),
-    UniProt ID(s), and PDB accession(s).
-
-    :param row: tr element from CAZy family protein page
-    :param family_name: str, name of CAZy family
-
-    Return Protein instance.
-    """
-    # retrieve list of cells ('td' elements) in row
-    tds = list(row.find_all("td"))
-
-    protein_name = tds[0].contents[0].strip()
-    source_organism = tds[2].a.get_text()
-    ec_numbers = []
-    links = {}
-
-    all_links = None
-    try:
-        all_links = tds[1].find_all("a")
-    except TypeError:  # raised when no EC numbers are listed
-        pass
-
-    if all_links is not None:
-        for link in all_links:
-            ec_numbers.append(link.text)
-    else:
-        ec_numbers = [np.nan]
-
-    # test for len(tds[x].contents) in case there is no link,
-    # the check of .name then ensures link is captured
-    if len(tds[3].contents) and tds[3].contents[0].name == "a":
-        links["GenBank"] = [f"{_.get_text()} {_['href']}" for _ in tds[3].contents if
-                            _.name == "a"]
-    if len(tds[4].contents) and tds[4].contents[0].name == "a":
-        links["UniProt"] = [f"{_.get_text()} {_['href']}" for _ in tds[4].contents if
-                            _.name == "a"]
-    if len(tds[5].contents) and tds[5].contents[0].name == "a":
-        links["PDB/3D"] = [f"{_.get_text()} {_['href']}" for _ in tds[5].contents if
-                           _.name == "a"]
-
-    # Retrieve GenBank accession synonms
-    try:
-        if len(links["GenBank"]) != 0:
-            # retrieve HTML link GenBank accession (CAZy only hyperlinks the first listed accession)
-            key = links["GenBank"][0]
-            key = key.split(" ")[0]  # remove the URL address
-
-            # retrieve all GenBank accessions listed in td element
-            all_genbank_accessions = tds[3].get_text(separator=" ")
-            all_genbank_accessions = all_genbank_accessions.split(" ")
-
-            if key in all_genbank_accessions:
-                all_genbank_accessions.remove(key)
-
-            # remove duplicates
-            all_genbank_accessions = list(dict.fromkeys(all_genbank_accessions))
-
-            if len(all_genbank_accessions) == 0:
-                genbank_synonyms = None
-            else:
-                genbank_synonyms = {key: all_genbank_accessions}
-        else:
-            genbank_synonyms = None
-    except KeyError:
-        genbank_synonyms = None
-
-    return Protein(protein_name, family_name, ec_numbers, source_organism, links, genbank_synonyms)
-
-
-def browser_decorator(func):
-    """Decorator to retry the wrapped function up to 'retries' times."""
-
-    def wrapper(*args, retries=10, **kwargs):
-        tries, success, err = 0, False, None
-        while not success and (tries < retries):
-            try:
-                response = func(*args, **kwargs)
-            except (
-                ConnectionError,
-                HTTPError,
-                OSError,
-                MissingSchema,
-                RequestError,
-            ) as err_message:
-                success = False
-                response = None
-                err = err_message
-            if response is not None:  # response was successful
-                success = True
-            # if response from webpage was not successful
-            tries += 1
-            time.sleep(10)
-        if (not success) or (response is None):
-            return [None, err]
-        else:
-            return [response, None]
-
-    return wrapper
-
-
-@browser_decorator
-def get_page(url):
-    """Create browser and use browser to retrieve page for given URL.
-
-    :param url: str, url to webpage
+        if len(class_urls) == 0:
+            logger.error("Failed to retrieve URLs to CAZy class pages.\nTerminating program")
+            sys.exit(1)
+    except TypeError:  # rased when class_pages is None
+        logger.error("Failed to retrieve URLs to CAZy class pages.\nTerminating program")
+        sys.exit(1)
 
-    Return browser response object (the page).
-    """
-    # create browser object
-    browser = mechanicalsoup.Browser()
-    # create response object
-    page = browser.get(url)
-    page = page.soup
+    # add storing the number of times an attempt to scrape the class page as been performed
+    index = 0
+    for index in range(len(class_urls)):
+        # list structure [class_url, number_of_tried_scrapes]
+        class_urls[index] = [class_urls[index], 0]
 
-    return page
+    return class_urls
 
 
 if __name__ == "__main__":
diff --git a/scraper/crawler/__init__.py b/scraper/crawler/__init__.py
new file mode 100644
index 00000000..7c829474
--- /dev/null
+++ b/scraper/crawler/__init__.py
@@ -0,0 +1,532 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Author:
+# Emma E. M. Hobbs
+
+# Contact
+# eemh1@st-andrews.ac.uk
+
+# Emma E. M. Hobbs,
+# Biomolecular Sciences Building,
+# University of St Andrews,
+# North Haugh Campus,
+# St Andrews,
+# KY16 9ST
+# Scotland,
+# UK
+
+# The MIT License
+"""Module for crawling through the CAZy website and parsing HTML from the CAZy website.
+
+:func get_cazy_family_urls: retrieve URLs to families on CAZy class summary page
+:func get_subfamily_links: retrieve URLs to subfamilies on CAZy class summary page
+:func parse_family: build Family class object to represent CAZy family
+:func parse_family_pages: retrieve all URLs to pages containing protein records for CAZy family
+:func parse_proteins: retrieve protein records from protein table page
+:func row_to_protein: parse the protein record to build a Protein class object
+"""
+
+
+import re
+import sys
+import time
+
+import numpy as np
+
+from collections import defaultdict
+from tqdm import tqdm
+from requests.exceptions import ConnectionError, MissingSchema
+from urllib3.exceptions import HTTPError, RequestError
+
+import mechanicalsoup
+
+
+class Protein:
+    """A single protein.
+
+    Each protein has a name, source organism (source), and links to external databases. The links to
+    external databases are stored in a dictionary, keyed by the external database name ('str') with
+    'list' values becuase there may be multiple links per database.
+
+    Multiple 'synonym' GenBank accession numbers maybe listed for a single protein. CAZy only
+    hyperlinks the first listed accession number. This accession is the one listed for the protein,
+    because is presumed to be the accession used by CAZy in their classification. All other listed
+    GenBank accessions are regarded as synonyms, including for example splice variants and identical
+    protein sequence submissions.
+    """
+
+    def __init__(self, name, family, ec, source, links=None, genbank_synonyms=None):
+        self.name = name
+        self.family = family
+        self.ec = ec
+        self.source = source
+        if links is None:
+            self.links = defaultdict(list)
+        else:
+            self.links = links
+        self.genbank_synonyms = genbank_synonyms
+
+    def __str__(self):
+        """Create representative string of class object"""
+        return f"{self.name} ({self.family} {self.source}): links to {self.links.keys()}"
+
+    def __repr__(self):
+        """Create representative object"""
+        return (
+            f"<Protein: {id(self)}: {self.name}, {self.family} "
+            f"({self.source}), {len(self.links)} to external databases>"
+        )
+
+    def get_protein_dict(self):
+        """Return a dictionary containing all the data of the protein."""
+        protein_dict = {"Protein_name": [self.name], "CAZy_family": [self.family]}
+
+        if len(self.ec) == 0:
+            protein_dict["EC#"] = [np.nan]
+        elif len(self.ec) == 1:
+            protein_dict["EC#"] = self.ec
+        else:
+            ec_string = "\n".join(self.ec)
+            protein_dict["EC#"] = [ec_string]
+
+        protein_dict["Source_organism"] = [self.source]
+
+        if type(self.links) is dict:
+            for database in ["GenBank", "UniProt", "PDB/3D"]:
+                try:
+                    if len(self.links[database]) == 1:
+                        protein_dict[database] = self.links[database]
+                    else:
+                        accession_string = ",\n".join(self.links[database])
+                        protein_dict[database] = [accession_string]
+                except KeyError:
+                    protein_dict[database] = [np.nan]
+        else:
+            for database in ["GenBank", "UniProt", "PDB/3D"]:
+                protein_dict[database] = [np.nan]
+        return protein_dict
+
+
+class Family:
+    """A single CAZy family."""
+
+    members = set()  # holds Protein instances
+
+    def __init__(self, name, cazy_class):
+        self.name = name
+        self.cazy_class = cazy_class
+
+    def __str__(self):
+        return f"CAZy family {self.name}: {len(self.members)} protein members"
+
+    def __repr__(self):
+        return f"<Family: {id(self)}: {self.name}, {len(self.members)} protein members"
+
+    def get_proteins(self):
+        """Return a list of all protein members of the CAZy family."""
+        return self.members
+
+    def get_family_name(self):
+        """Return family name"""
+        return self.name
+
+
+def get_cazy_class_urls(cazy_home, excluded_classes, max_tries, logger):
+    """Returns a list of CAZy class main/home page URLs for each specified class as the CAZy site.
+
+    :param cazy_url: str, URL to the CAZy home page.
+    :param excluded_classes: list, list of CAZy classes not to be scraped
+    :param max_tries: int, maximum number of times to try scrape if errors are encountered
+    :param logger: logger object
+
+    Return list of URLs.
+    """
+    logger.info("Retrieving URLs to summary CAZy class pages")
+
+    # define items to be excluded from returned class list, ALWAYS exlide links to genomes
+    if excluded_classes is not None:
+        exclusions = tuple(["<strong>Genomes</strong>"] + excluded_classes)
+    else:
+        exclusions = "<strong>Genomes</strong>"
+
+    home_page = [None, None]
+    tries = 0  # number of the try/attempt
+
+    while (home_page[0] is None) and (tries < max_tries):
+        home_page = get_page(cazy_home)
+
+        if (home_page[0] is None) and (tries < max_tries):
+            logger.error(
+                f"Failed to connect to CAZy homepage after 10 attempts,\n"
+                f"for attempt# {(tries+1)}/{max_tries}\n"
+                "The following error was raised:\n"
+                f"{home_page[1]}\n"
+                f"Reattempting for attempt# {(tries+2)} in 10s."
+            )
+            time.sleep(10)
+            tries += 1
+
+    if home_page[0] is None:
+        logger.error(
+            (
+                "Failed to connect to CAZy home-page after multiple attempts.\n"
+                "The following error was raised:\n"
+                f"{home_page[1]}"
+                "Could not retrieve URLs to CAZy classes.\n"
+                "Check the network connection.\n"
+                "Terminating program."
+            )
+        )
+        sys.exit(1)
+
+    try:
+        return [
+            f"{cazy_home}/{_['href']}"
+            for _ in home_page[0].find_all("a", {"class": "spip_out"})
+            if (not _["href"].startswith("http")) and (str(_.contents[0]) not in exclusions)
+        ]
+    except AttributeError:  # raise if can't find results with find_all("a", {"class": "spip_out"})
+        return None
+
+
+def get_cazy_family_urls(class_url, cazy_home, class_name, args, logger):
+    """Retrieve all protein members of each CAZy family within the given CAZy class.
+
+    :param class_url: str, URL to the CAZy class
+    :param cazy_home: str, URL to CAZy home page
+    :param class_name: str, name of CAZy class
+    :param args: args parser object
+    :param logger: logger object
+
+    Returns list of URLs to family pages.
+    """
+    logger.info(f"Retrieving URLs to families under {class_name}")
+
+    # scrape the class page
+    class_page = get_page(class_url)
+
+    if class_page[0] is None:
+        logger.error(
+            (
+                f"Failed to connect to {class_url} after 10 attempts.\n"
+                "The following error was raised:\n"
+                f"{class_page[1]}"
+                "Could not retrieve URLs to CAZy famileis for this class.\n"
+                "This class will be skipped during the scraping process."
+            )
+        )
+        return None
+
+    # retrieve the <h3> element that titles the div section containing the tables of family links
+    family_h3_element = [
+        _
+        for _ in class_page[0].find_all("h3", {"class": "spip"})
+        if str(_.contents[0]).strip() == "Tables for Direct Access"
+    ][0]
+
+    # retrieve all tables within the parent div section of the <h3> element
+    tables = family_h3_element.parent.find_all("table")
+
+    # tables[0] is the table containing links to CAZy families
+    # tables[1] is the table containing the link to unclassified proteins
+
+    family_urls = family_urls = [f"{cazy_home}/{_['href']}" for _ in tables[0].find_all("a")]
+    family_urls.append(f"{cazy_home}/{tables[1].a['href']}")
+    if args.subfamilies is True:
+        subfam_urls = get_subfamily_links(family_h3_element, cazy_home, logger)
+        if subfam_urls is not None:
+            family_urls += subfam_urls
+
+    if family_urls is not None:
+        # add storage of number of attempts to scrape the Family _all page
+        index = 0
+        for index in range(len(family_urls)):
+            family_urls[index] = [family_urls[index], 0]
+
+    return family_urls
+
+
+def get_subfamily_links(family_h3_element, cazy_home, logger):
+    """Retrieve URL links to CAZy subfamilies.
+
+    :param family_h3_element: bs4.element.Tag, h3 element titling the page div
+    :param cazy_home: str, URL to CAZy home_page
+    :param logger: logger object
+
+    Return list of URLs to subfamilies.
+    """
+    parent_div = family_h3_element.parent
+    all_links = parent_div.find_all("a")
+
+    pattern = re.compile(r"\D+?\d+?_\d+?\.html")
+
+    urls = []  # empty list to store subfamily URLs
+
+    for link in all_links:
+        try:
+            search_result = re.search(pattern, link["href"])
+            urls.append(f"{cazy_home}/{search_result.group()}")
+        except (KeyError, AttributeError) as error:
+            # KeyError raised if link does not have ['href']
+            # AttributeError error raised if search_result is None becuase not subfam link
+            pass
+
+    if len(urls) == 0:
+        return
+    else:
+        return urls
+
+
+def parse_family(family_url, family_name, cazy_home, logger):
+    """Returns a Family object with Protein members, scraped from CAZy.
+
+    :param family_url: str, URL to CAZy family summary page
+    :param family_name: str, name of CAZy family
+    :param cazy_home: str, URL to CAZy home page
+    :param logger: logger object
+
+    Return list of Family object encapsulating data from the passed family
+    summary page, and an error code if failed to scrape the family 'all' page.
+    """
+    logger.info(f"Starting retrieval of proteins for {family_name} from {family_url}")
+
+    # retrieve class from family name
+    pattern = re.compile(r"\D+")
+    search_result = re.match(pattern, family_name)
+    try:
+        cazy_class = search_result.group()
+    except AttributeError:
+        logger.warning(
+            f"Incorrect formating of the family name {family_name}\n"
+            "Not scrapping this family"
+        )
+        family = Family(family_name, "Incorrect family name format")
+        family.members = set()
+        return [family, "Incorrect formatting of family name"]
+
+    family = Family(family_name, cazy_class)
+    family.members = set()
+
+    # compile URL to first family page of protein records
+    first_pagination_url = family_url.replace(".html", "_all.html")
+
+    # check url formating
+    try:
+        re.match(
+            r"http://www.cazy.org/\D{2,3}(\d+|\d+_\d+)_all.html", first_pagination_url
+        ).group()
+    except AttributeError:
+        logger.warning(
+            f"Incorrect formatting of first protein table page URL: {first_pagination_url}\n"
+            "Will not try and connect to this URL."
+        )
+        return [family, "Incorrect first protein table page URL formating"]
+
+    first_pagination_page = get_page(first_pagination_url)
+
+    if first_pagination_page[0] is None:
+        logger.warning(
+            (
+                f"Could not connect to {family_url}\n"
+                "The following error was raised:\n"
+                f"{first_pagination_page[1]}"
+                f"No protein records for CAZy family {family_name} will be retried."
+            )
+        )
+        return [family, first_pagination_page[1]]
+
+    protein_page_urls = get_protein_page_urls(
+        first_pagination_url,
+        first_pagination_page[0],
+        cazy_home,
+    )
+
+    for protein in tqdm(
+        (y for x in (parse_proteins(url, family_name, logger) for url in protein_page_urls) for y in x),
+        total=len(protein_page_urls),
+        desc=f"Scraping protein pages for {family_name}",
+    ):
+        family.members.add(protein)
+
+    return [family, None]
+
+
+def get_protein_page_urls(first_pagination_url, first_pagination_page, cazy_home):
+    """Retrieve the URLs to all pages containing proteins for the current working family.
+
+    :param first_pagination_url: str, URL to first page contaiing proteins
+    :param first_pagination_page: BS4 object, first page containing proteins
+    :param cazy_home: str, URL of CAZy homepage
+
+    Return list of URLs.
+    """
+    protein_page_urls = [first_pagination_url]
+
+    # retrieve the URL to the final page of protein records in the pagination listing
+    try:
+        last_pagination_url = first_pagination_page.find_all("a", {"class": "lien_pagination", "rel": "nofollow"})[-1]
+    except IndexError:  # there is no pagination; a single-query entry
+        last_pagination_url = None
+
+    if last_pagination_url is not None:
+        url_prefix = last_pagination_url["href"].split("PRINC=")[0] + "PRINC="
+        last_princ_no = int(last_pagination_url["href"].split("PRINC=")[-1].split("#pagination")[0])
+        url_suffix = "#pagination" + last_pagination_url["href"].split("#pagination")[-1]
+
+        # Build list of urls to all pages in the pagination listing, increasing the PRINC increment
+        protein_page_urls.extend(
+            [f"{cazy_home}/{url_prefix}{_}{url_suffix}" for _ in range(1000, last_princ_no + 1000, 1000)]
+        )
+
+    return protein_page_urls
+
+
+def parse_proteins(protein_page_url, family_name, logger):
+    """Returns generator of Protein objects for all protein rows on a single CAZy family page.
+
+    :param protein_page_url, str, URL to the CAZy family page containing protein records
+    :param family_name: str, name of CAZy family
+    :param logger: logger object
+
+    Return generator object.
+    """
+    logger.info(f"Retrieving proteins from {protein_page_url}")
+
+    protein_page = get_page(protein_page_url)
+
+    if protein_page[0] is None:
+        logger.warning(
+            (
+                f"Could not connect to {protein_page_url}\n"
+                "The following error was raised:\n"
+                f"{protein_page[1]}\n"
+                f"No protein records from this page will be retried."
+            )
+        )
+        return
+
+    # retrieve protein record table
+    protein_table = protein_page[0].find_all("table", {"class": "listing"})[0]
+    protein_rows = [
+        _ for _ in protein_table.descendants if (_.name == "tr") and ("id" not in _.attrs) and ("class" not in _.attrs)
+    ]
+
+    for row in protein_rows:
+        yield row_to_protein(row, family_name)
+
+
+def row_to_protein(row, family_name):
+    """Returns a Protein object representing a single protein row from a CAZy family protein page.
+
+    Each row, in order, contains the protein name, EC number, source organism, GenBank ID(s),
+    UniProt ID(s), and PDB accession(s).
+
+    :param row: tr element from CAZy family protein page
+    :param family_name: str, name of CAZy family
+
+    Return Protein instance.
+    """
+    # retrieve list of cells ('td' elements) in row
+    tds = list(row.find_all("td"))
+
+    protein_name = tds[0].contents[0].strip()
+    source_organism = tds[2].a.get_text()
+    ec_numbers = []
+    links = {}
+
+    all_links = None
+    try:
+        all_links = tds[1].find_all("a")
+    except TypeError:  # raised when no EC numbers are listed
+        pass
+
+    if all_links is not None:
+        for link in all_links:
+            ec_numbers.append(link.text)
+    else:
+        ec_numbers = [np.nan]
+
+    # test for len(tds[x].contents) in case there is no link,
+    # the check of .name then ensures link is captured
+    if len(tds[3].contents) and tds[3].contents[0].name == "a":
+        links["GenBank"] = [f"{_.get_text()} {_['href']}" for _ in tds[3].contents if _.name == "a"]
+    if len(tds[4].contents) and tds[4].contents[0].name == "a":
+        links["UniProt"] = [f"{_.get_text()} {_['href']}" for _ in tds[4].contents if _.name == "a"]
+    if len(tds[5].contents) and tds[5].contents[0].name == "a":
+        links["PDB/3D"] = [f"{_.get_text()} {_['href']}" for _ in tds[5].contents if _.name == "a"]
+
+    # Retrieve GenBank accession synonms
+    try:
+        if len(links["GenBank"]) != 0:
+            # retrieve HTML link GenBank accession (CAZy only hyperlinks the first listed accession)
+            key = links["GenBank"][0]
+            key = key.split(" ")[0]  # remove the URL address
+
+            # retrieve all GenBank accessions listed in td element
+            all_genbank_accessions = tds[3].get_text(separator=" ")
+            all_genbank_accessions = all_genbank_accessions.split(" ")
+
+            if key in all_genbank_accessions:
+                all_genbank_accessions.remove(key)
+
+            # remove duplicates
+            all_genbank_accessions = list(dict.fromkeys(all_genbank_accessions))
+
+            if len(all_genbank_accessions) == 0:
+                genbank_synonyms = None
+            else:
+                genbank_synonyms = {key: all_genbank_accessions}
+        else:
+            genbank_synonyms = None
+    except KeyError:
+        genbank_synonyms = None
+
+    return Protein(protein_name, family_name, ec_numbers, source_organism, links, genbank_synonyms)
+
+
+def browser_decorator(func):
+    """Decorator to retry the wrapped function up to 'retries' times."""
+
+    def wrapper(*args, retries=10, **kwargs):
+        tries, success, err = 0, False, None
+        while not success and (tries < retries):
+            try:
+                response = func(*args, **kwargs)
+            except (
+                ConnectionError,
+                HTTPError,
+                OSError,
+                MissingSchema,
+                RequestError,
+            ) as err_message:
+                success = False
+                response = None
+                err = err_message
+            if response is not None:  # response was successful
+                success = True
+            # if response from webpage was not successful
+            tries += 1
+            time.sleep(10)
+        if (not success) or (response is None):
+            return [None, err]
+        else:
+            return [response, None]
+
+    return wrapper
+
+
+@browser_decorator
+def get_page(url):
+    """Create browser and use browser to retrieve page for given URL.
+
+    :param url: str, url to webpage
+
+    Return browser response object (the page).
+    """
+    # create browser object
+    browser = mechanicalsoup.Browser()
+    # create response object
+    page = browser.get(url)
+    page = page.soup
+
+    return page
diff --git a/scraper/file_io/__init__.py b/scraper/file_io/__init__.py
index 41273740..265ac10c 100644
--- a/scraper/file_io/__init__.py
+++ b/scraper/file_io/__init__.py
@@ -100,9 +100,9 @@ def parse_configuration(file_io_path, args, logger):
             config_dict = yaml.full_load(fh)
     except FileNotFoundError:
         logger.warning(
-            "Could noot find configuration file.\n"
+            "Could not find configuration file.\n"
             "Make sure path to the configuration file is correct\n"
-            "Scrapping will continue using defaul scraper configuration\n"
+            "Scrapping will continue using default scraper configuration\n"
         )
         return None, None, cazy_dict
 
@@ -192,27 +192,57 @@ def write_out_df(dataframe, df_name, outdir, logger, force):
 
     Return nothing.
     """
-    # build output path
-    output_path = outdir / f"{df_name}.csv"
-
-    logger.info("Checking if output directory for dataframe already exists")
-    if output_path.exists():
-        if force is False:
-            logger.warning(
-                (
-                    "Specified directory for dataframe already exists.\n"
-                    "Exiting writing out dataframe."
+    if outdir is not sys.stdout:
+        # build output path
+        output_path = outdir / f"{df_name}.csv"
+
+        logger.info("Checking if output directory for dataframe already exists")
+        if output_path.exists():
+            if force is False:
+                logger.warning(
+                    (
+                        "Specified directory for dataframe already exists.\n"
+                        "Exiting writing out dataframe."
+                    )
                 )
-            )
-            return ()
-        else:
-            logger.warning(
-                (
-                    "Specified directory for dataframe already exists.\n"
-                    "Forced overwritting enabled."
+                return ()
+            else:
+                logger.warning(
+                    (
+                        "Specified directory for dataframe already exists.\n"
+                        "Forced overwritting enabled."
+                    )
                 )
-            )
 
-    logger.info("Writing out species dataframe to directory")
-    dataframe.to_csv(output_path)
+        logger.info("Writing out species dataframe to directory")
+        dataframe.to_csv(output_path)
+
+    else:
+        logger.info(f"Output dataframe: {df_name}")
+        dataframe.to_csv(outdir)
+
+    return
+
+
+def write_out_failed_scrapes(failed_urls, time_stamp, args, logger):
+    """Write out the URLs for which a connection to CAZy failed.
+
+    :param failed_urls: list, contains the URL and reason for the failed scrape
+    :param args: cmd args parser
+    :param logger: logger object
+
+    Return nothing.
+    """
+    if args.output is not sys.stdout:
+        output_path = args.output / f"failed_cazy_scrapes_{time_stamp}.txt"
+
+        with open(output_path, "a") as fh:
+            for url in failed_urls:
+                fh.write(f"{url}\n")
+
+    else:
+        logger.error("The following items were not scraped:")
+        for url in failed_urls:
+            logger.error(url)
+
     return
diff --git a/scraper/parse/__init__.py b/scraper/parse/__init__.py
index 7b03f7d2..03d35c36 100644
--- a/scraper/parse/__init__.py
+++ b/scraper/parse/__init__.py
@@ -27,8 +27,6 @@
 
 import pandas as pd
 
-from datetime import datetime
-
 from Bio import Entrez, SeqIO
 from Bio.PDB import PDBList
 from tqdm import tqdm
@@ -36,7 +34,7 @@
 from scraper.file_io import write_out_df
 
 
-def proteins_to_dataframe(families, args, logger):
+def proteins_to_dataframe(families, time_stamp, args, logger):
     """Write protein members of CAZy families to a pandas dataframe.
 
     Duplicate rows are removed. These rows must be identical across the entire row, not only share
@@ -80,8 +78,6 @@ def proteins_to_dataframe(families, args, logger):
                 if genbank_synonyms is not None:
                     all_genbank_synonyms.update(genbank_synonyms)
 
-    time_stamp = datetime.now().strftime("%Y-%m-%d--%H-%M-%S")
-
     if args.subfamilies is True:
         subfam = "_incld_subfams"
     else:
diff --git a/scraper/scraper_config.yaml b/scraper/scraper_config.yaml
index f3cd6f74..af791d30 100644
--- a/scraper/scraper_config.yaml
+++ b/scraper/scraper_config.yaml
@@ -7,9 +7,9 @@
 # https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html 
 classes:  # classes from which all proteins will be retrieved
 Glycoside Hydrolases (GHs):
-  - "GH4"
 GlycosylTransferases (GTs):
 Polysaccharide Lyases (PLs):
+  - "PL28"
 Carbohydrate Esterases (CEs):
 Auxiliary Activities (AAs):
 Carbohydrate-Binding Modules (CBMs):
\ No newline at end of file
diff --git a/scraper/utilities/__init__.py b/scraper/utilities/__init__.py
index d5ca7916..45c5e062 100644
--- a/scraper/utilities/__init__.py
+++ b/scraper/utilities/__init__.py
@@ -137,6 +137,15 @@ def build_parser(argv: Optional[List] = None):
         help="Enable downloading of protein structures in XXXX format from PDB",
     )
 
+    # Add option to enable number of times to retry scraping
+    parser.add_argument(
+        "-r",
+        "--retries",
+        type=int,
+        default=0,
+        help="Number of times to retry scraping a family or class page if error encountered",
+    )
+
     # Add option to enable retrieval of subfamilies
     parser.add_argument(
         "-s",
diff --git a/setup.py b/setup.py
index 50fa1ae2..a6c233e0 100644
--- a/setup.py
+++ b/setup.py
@@ -50,7 +50,15 @@
     platforms="Posix, MacOS X",
     url="https://github.com/HobnobMancer/cazy_webscraper",
     entry_points={"console_scripts": ["cazy_webscraper.py = scraper.cazy_webscraper:main"]},
-    install_requires=["biopython>=1.76", "pandas>=1.0.3", "requests", "mechanicalsoup", "tqdm", "pyyaml"],
+    install_requires=[
+        "biopython>=1.76",
+        "mechanicalsoup",
+        "pandas>=1.0.3",
+        "pyyaml",
+        "requests",
+        "sqlalchemy==1.3.20",
+        "tqdm",
+    ],
     packages=setuptools.find_packages(),
     package_data={
         "Conda microenvironment": ["environment.yml"],


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index b8d64cc6..a7b61a04 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -22,6 +22,7 @@
 
 import logging
 
+from datetime import datetime
 from pathlib import Path
 
 import pytest
@@ -54,3 +55,9 @@ def cazy_home_url():
 def cazy_dictionary(test_input_dir):
     dict_path = test_input_dir / "cazy_dictionary.json"
     return dict_path
+
+
+@pytest.fixture
+def time_stamp():
+    time_stamp = datetime.now().strftime("%Y-%m-%d--%H-%M-%S")  # used in naming files
+    return time_stamp
diff --git a/tests/test_crawler.py b/tests/test_crawler.py
new file mode 100644
index 00000000..aca7e425
--- /dev/null
+++ b/tests/test_crawler.py
@@ -0,0 +1,568 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# Author:
+# Emma E. M. Hobbs
+
+# Contact
+# eemh1@st-andrews.ac.uk
+
+# Emma E. M. Hobbs,
+# Biomolecular Sciences Building,
+# University of St Andrews,
+# North Haugh Campus,
+# St Andrews,
+# KY16 9ST
+# Scotland,
+# UK
+
+# The MIT License
+
+"""Tests the script scraper.crawler submodule which crawls through the CAZy database.
+
+These test are intened to be run from the root of the repository using:
+pytest -v
+"""
+
+import types
+import pytest
+
+
+from argparse import Namespace
+from requests.exceptions import MissingSchema
+
+from bs4 import BeautifulSoup
+
+from scraper import crawler
+
+
+@pytest.fixture
+def input_dir(test_input_dir):
+    dir_path = test_input_dir / "test_inputs_crawler"
+    return dir_path
+
+
+@pytest.fixture
+def cazy_home_page(input_dir):
+    file_path = input_dir / "cazy_homepage.html"
+    return file_path
+
+
+@pytest.fixture
+def cazy_home_no_spip(input_dir):
+    file_path = input_dir / "cazy_homepage_no_spip_out.html"
+    return file_path
+
+
+@pytest.fixture
+def args_datasplit_family():
+    argsdict = {
+        "args": Namespace(
+            data_split="family",
+            subfamilies=True,
+        )
+    }
+    return argsdict
+
+
+@pytest.fixture
+def cazy_class_page(input_dir):
+    file_path = input_dir / "cazy_classpage.html"
+    return file_path
+
+
+@pytest.fixture
+def family_urls(input_dir):
+    file_path = input_dir / "test_family_urls.txt"
+
+    with open(file_path, "r") as fh:
+        fam_lines = fh.read().splitlines()
+
+    fam_list = []
+    for line in fam_lines:
+        fam_list.append([line, 0])
+
+    return fam_list
+
+
+@pytest.fixture
+def family_h3_element(cazy_class_page):
+    with open(cazy_class_page) as fp:
+        soup = BeautifulSoup(fp, features="lxml")
+
+    return [_ for _ in
+            soup.find_all("h3", {"class": "spip"}) if
+            str(_.contents[0]) == "Tables for Direct Access"][0]
+
+
+@pytest.fixture
+def subfamily_urls(input_dir):
+    file_path = input_dir / "subfamily_urls.txt"
+    with open(file_path, "r") as fh:
+        fam_string = fh.read()
+    fam_string = fam_string[1:-1]
+    fam_string = fam_string.replace("'", "")
+    fam_list = fam_string.split(", ")
+    return fam_list
+
+
+@pytest.fixture
+def no_subfam_h3_element(input_dir):
+    file_path = input_dir / "cazy_classpage_no_subfams.html"
+    with open(file_path) as fp:
+        soup = BeautifulSoup(fp, features="lxml")
+
+    return [_ for _ in
+            soup.find_all("h3", {"class": "spip"}) if
+            str(_.contents[0]) == "Tables for Direct Access"][0]
+
+
+@pytest.fixture
+def protein_list():
+    return [
+        crawler.Protein("protein_name", "GH1", "1.2.3.4", "organism"),
+        crawler.Protein(
+            "protein",
+            "GH1",
+            "",
+            "organism",
+            {"GenBank": ["link1"], "UniProt": ["link2"], "PDB": ["link3"]},
+        ),
+    ]
+
+
+@pytest.fixture
+def protein_gen(protein_list):
+    return (_ for _ in protein_list)
+
+
+@pytest.fixture
+def gh1_page(input_dir):
+    file_path = input_dir / "cazy_gh1_page.html"
+    return file_path
+
+
+@pytest.fixture
+def gh147_page(input_dir):
+    file_path = input_dir / "cazy_gh147_page.html"
+    return file_path
+
+
+@pytest.fixture
+def no_links_page(input_dir):
+    file_path = input_dir / "cazy_no_links.html"
+    return file_path
+
+
+@pytest.fixture
+def protein_without_ec(input_dir):
+    file_path = input_dir / "protein_without_ec.html"
+    return file_path
+
+
+@pytest.fixture
+def protein_with_ec(input_dir):
+    file_path = input_dir / "protein_with_ec.html"
+    return file_path
+
+
+@pytest.fixture
+def protein_with_gb_synonyms(input_dir):
+    file_path = input_dir / "protein_with_genbank_synonyms.html"
+    return file_path
+
+
+@pytest.fixture
+def protein_with_no_gb(input_dir):
+    file_path = input_dir / "protein_no_genbank.html"
+    return file_path
+
+
+# test the classes Protein and Family
+
+
+def test_protein_get_protein_dict():
+    """Test the Protein class __str__ and __repr__."""
+    protein = crawler.Protein(
+        "protein_name",
+        "GH1",
+        ["1.2.3.4"],
+        "organism",
+        {"GenBank": ["link1", "linka", "linkb"], "UniProt": ["link2"], "PDB/3D": ["link3"]},
+    )
+
+    protein_dict = protein.get_protein_dict()
+
+    expected_protein_dict = {
+        'Protein_name': ['protein_name'],
+        'CAZy_family': ['GH1'],
+        'EC#': ['1.2.3.4'],
+        'Source_organism': ['organism'],
+        'GenBank': ['link1,\nlinka,\nlinkb'],
+        'UniProt': ['link2'],
+        'PDB/3D': ['link3'],
+    }
+
+    assert expected_protein_dict == protein_dict
+
+
+def test_family_get_name():
+    """Tests get family name for Family."""
+    family = crawler.Family("GH1", "Glycoside_Hydrolases(GH)")
+
+    family_name = family.get_family_name()
+
+    exepected_name = "GH1"
+
+    assert exepected_name == family_name
+
+
+# test get_cazy_class_urls
+
+
+def test_get_class_urls_exclusions_none(cazy_home_url, cazy_home_page, null_logger, monkeypatch):
+    """Test get_cazy_class_urls when excluded_classess is None."""
+    with open(cazy_home_page, "r") as fp:
+        home_page = BeautifulSoup(fp, features="lxml")
+
+        def mock_get_home_page(*args, **kwargs):
+            return [home_page, None]
+
+    monkeypatch.setattr(crawler, "get_page", mock_get_home_page)
+
+    expected_result = [
+        'http://www.cazy.org/Glycoside-Hydrolases.html',
+        'http://www.cazy.org/GlycosylTransferases.html',
+        'http://www.cazy.org/Polysaccharide-Lyases.html',
+        'http://www.cazy.org/Carbohydrate-Esterases.html',
+        'http://www.cazy.org/Auxiliary-Activities.html',
+        'http://www.cazy.org/Carbohydrate-Binding-Modules.html',
+    ]
+
+    assert expected_result == crawler.get_cazy_class_urls(
+        cazy_home_url,
+        None,
+        1,
+        null_logger,
+    )
+
+
+def test_get_class_urls_fail(cazy_home_url, null_logger, monkeypatch):
+    """Test get_cazy_class_urls home_page not returned"""
+
+    def mock_get_home_page(*args, **kwargs):
+        return [None, "error"]
+
+    monkeypatch.setattr(crawler, "get_page", mock_get_home_page)
+
+    with pytest.raises(SystemExit) as pytest_wrapped_e:
+        crawler.get_cazy_class_urls(cazy_home_url, None, 1, null_logger)
+    assert pytest_wrapped_e.type == SystemExit
+
+
+def test_get_class_urls_exclusions_given(
+    cazy_home_url,
+    cazy_home_page,
+    null_logger,
+    monkeypatch,
+):
+    """Test get_cazy_class_urls when excluded_classess is not None."""
+    with open(cazy_home_page) as fp:
+        soup = BeautifulSoup(fp, features="lxml")
+
+    exclusions = ["<strong>Glycoside Hydrolases (GHs)</strong>"]
+
+    def mock_get_home_page(*args, **kwargs):
+        return [soup, None]
+
+    monkeypatch.setattr(crawler, "get_page", mock_get_home_page)
+
+    expected_result = [
+        'http://www.cazy.org/GlycosylTransferases.html',
+        'http://www.cazy.org/Polysaccharide-Lyases.html',
+        'http://www.cazy.org/Carbohydrate-Esterases.html',
+        'http://www.cazy.org/Auxiliary-Activities.html',
+        'http://www.cazy.org/Carbohydrate-Binding-Modules.html',
+    ]
+
+    assert expected_result == crawler.get_cazy_class_urls(
+        cazy_home_url,
+        exclusions,
+        1,
+        null_logger,
+    )
+
+
+def test_get_class_urls_attribute(
+    cazy_home_url,
+    cazy_home_no_spip,
+    null_logger,
+    monkeypatch,
+):
+    """Test get_cazy_class_urls when attribute error is raised."""
+    with open(cazy_home_no_spip) as fp:
+        soup = fp.read()
+
+    exclusions = ["<strong>Glycoside Hydrolases (GHs)</strong>"]
+
+    def mock_get_home_page(*args, **kwargs):
+        return [soup, None]
+
+    monkeypatch.setattr(crawler, "get_page", mock_get_home_page)
+
+    assert None is crawler.get_cazy_class_urls(
+        cazy_home_url,
+        exclusions,
+        1,
+        null_logger,
+    )
+
+
+# test get_cazy_family_urls
+
+
+def test_get_family_urls_fail(args_datasplit_family, null_logger, monkeypatch):
+    """Test get_cazy_family_urls when no page is returned."""
+
+    def mock_get_page(*args, **kwargs):
+        return [None, "error"]
+
+    monkeypatch.setattr(crawler, "get_page", mock_get_page)
+
+    assert None is crawler.get_cazy_family_urls(
+        "class_url",
+        "cazy_home_url",
+        "class_name",
+        args_datasplit_family["args"],
+        null_logger,
+    )
+
+
+def test_get_family_urls_success(
+    cazy_class_page,
+    args_datasplit_family,
+    family_urls,
+    null_logger,
+    monkeypatch,
+):
+    """Test get_cazy_family_urls when successful, and subfamilies is True."""
+    with open(cazy_class_page) as fp:
+        page = BeautifulSoup(fp, features="lxml")
+
+    def mock_get_page(*args, **kwargs):
+        return [page, None]
+
+    def mock_get_subfams(*args, **kwargs):
+        return []
+
+    monkeypatch.setattr(crawler, "get_page", mock_get_page)
+    monkeypatch.setattr(crawler, "get_subfamily_links", mock_get_subfams)
+
+    assert family_urls == crawler.get_cazy_family_urls(
+        "class_url",
+        "http://www.cazy.org",
+        "class_name",
+        args_datasplit_family["args"],
+        null_logger,
+    )
+
+
+# test get_subfamily_links
+
+
+def test_get_subfam_links_len_0(family_h3_element, subfamily_urls, null_logger):
+    """Test get_subfamily_links when no links are retrieved."""
+
+    assert subfamily_urls == crawler.get_subfamily_links(
+        family_h3_element,
+        "http://www.cazy.org",
+        null_logger,
+    )
+
+
+def test_get_subfam_links_urls(no_subfam_h3_element, null_logger):
+    """Test get_subfamily_links when urls are retrieved."""
+
+    assert None is crawler.get_subfamily_links(
+        no_subfam_h3_element,
+        "http://www.cazy.org",
+        null_logger,
+    )
+
+
+# test parse_family()
+
+
+def test_parse_family_fam_name(null_logger):
+    """Tests parse_family() when family name is incorrectly formated"""
+
+    crawler.parse_family(
+        "http://www.cazy.org/GH_test_1.html",
+        "GH_test_1",
+        "http://www.cazy.org",
+        null_logger,
+    )
+
+
+def test_parse_family_url(null_logger):
+    """Test parse_family() whwn the url is incorrectly formatted."""
+
+    crawler.parse_family(
+        "http://www.cazy.org/GH_test_1.html",
+        "GH1",
+        "http://www.cazy.org",
+        null_logger,
+    )
+
+
+def test_parse_family_no_page(null_logger, monkeypatch):
+    """Tests parse_family() when no page retrieved for first paginiation page of proteins"""
+
+    def mock_get_page(*args, **kwargs):
+        return [None, "error message"]
+
+    monkeypatch.setattr(crawler, "get_page", mock_get_page)
+
+    crawler.parse_family(
+        "http://www.cazy.org/GH1.html",
+        "GH1",
+        "http://www.cazy.org",
+        null_logger,
+    )
+
+
+def test_parse_family_success(protein_gen, null_logger, monkeypatch):
+    """Test parse_family() when successful."""
+
+    def mock_get_page(*args, **kwargs):
+        return ["first paginiation page", None]
+
+    def mock_get_urls(*args, **kwargs):
+        return ["http://www.cazy.org/GH1_all.html"]
+
+    def mock_parse_proteins(*args, **kwargs):
+        return protein_gen
+
+    monkeypatch.setattr(crawler, "get_page", mock_get_page)
+    monkeypatch.setattr(crawler, "get_protein_page_urls", mock_get_urls)
+    monkeypatch.setattr(crawler, "parse_proteins", mock_parse_proteins)
+
+    crawler.parse_family(
+        "http://www.cazy.org/GH1.html",
+        "GH1",
+        "http://www.cazy.org",
+        null_logger,
+    )
+
+
+# test get_protein_page_urls()
+
+
+def test_get_protein_page_urls_no_links(no_links_page):
+    """Test get_protein_page_urls() on page with no links."""
+    with open(no_links_page) as fp:
+        soup = BeautifulSoup(fp, features="lxml")
+
+    crawler.get_protein_page_urls("http://www.cazy.org/GH147_all.html", soup, "http://www.cazy.org")
+
+
+def test_get_protein_page_urls_no_pag(gh147_page):
+    """Test get_protein_page_urls() on page with no pagination of proteins."""
+    with open(gh147_page) as fp:
+        soup = BeautifulSoup(fp, features="lxml")
+
+    crawler.get_protein_page_urls("http://www.cazy.org/GH147_all.html", soup, "http://www.cazy.org")
+
+
+def test_get_protein_page_urls_pag(gh1_page):
+    """Test get_protein_page_urls() on page with pagination of proteins."""
+    with open(gh1_page) as fp:
+        soup = BeautifulSoup(fp, features="lxml")
+
+    crawler.get_protein_page_urls("http://www.cazy.org/GH1_all.html", soup, "http://www.cazy.org")
+
+
+# test parse_proteins
+
+
+def test_parse_proteins_none(null_logger, monkeypatch):
+    """Test parse_proteins when no page is returned."""
+
+    def mock_get_page(*args, **kwargs):
+        return [None, "error"]
+
+    monkeypatch.setattr(crawler, "get_page", mock_get_page)
+
+    assert True is isinstance(
+        crawler.parse_proteins("protein_url", "family", null_logger),
+        types.GeneratorType,
+    )
+
+
+def test_parse_proteins(gh147_page, null_logger, monkeypatch):
+    """Tests parse_proteins when successful."""
+    with open(gh147_page) as fp:
+        soup = BeautifulSoup(fp, features="lxml")
+
+    def mock_get_page(*args, **kwargs):
+        return soup
+
+    monkeypatch.setattr(crawler, "get_page", mock_get_page)
+
+    crawler.parse_proteins("protein_url", "family", null_logger)
+
+
+# test row_to_protein()
+
+
+def test_row_to_protein_no_ecs(protein_without_ec):
+    """Test row_to_protein when no EC#s are listed."""
+    with open(protein_without_ec) as fp:
+        row = BeautifulSoup(fp, features="lxml")
+
+    assert True is isinstance(
+        crawler.row_to_protein(row, "GH147"),
+        crawler.Protein
+    )
+
+
+def test_row_to_protein_ec(protein_with_ec):
+    """Test row_to_protein when EC#s are listed."""
+    with open(protein_with_ec) as fp:
+        row = BeautifulSoup(fp, features="lxml")
+
+    assert True is isinstance(
+        crawler.row_to_protein(row, "GH147"),
+        crawler.Protein
+    )
+
+
+def test_row_to_protein_gb_synoymns(protein_with_gb_synonyms):
+    """Test row_to_protein when protein has GenBank synonyms."""
+    with open(protein_with_gb_synonyms) as fp:
+        row = BeautifulSoup(fp, features="lxml")
+
+    assert True is isinstance(
+        crawler.row_to_protein(row, "GH147"),
+        crawler.Protein
+    )
+
+
+def test_row_to_protein_no_gb(protein_with_no_gb):
+    """Test row_to_protein when protein has no GenBank accession."""
+    with open(protein_with_no_gb) as fp:
+        row = BeautifulSoup(fp, features="lxml")
+
+    assert True is isinstance(
+        crawler.row_to_protein(row, "GH147"),
+        crawler.Protein
+    )
+
+
+# browser decorator and get_page
+
+
+def test_browser_decorator():
+    """Test browser_decorator to ensure proper handling if unsuccessful."""
+
+    result = crawler.get_page('www.caz!!!!!!!!y.org')
+    assert True == (result[0] is None) and (type(result[1]) is MissingSchema)
diff --git a/tests/test_file_io.py b/tests/test_file_io.py
index 52df7912..27190da5 100644
--- a/tests/test_file_io.py
+++ b/tests/test_file_io.py
@@ -159,6 +159,27 @@ def df_output_file(test_dir):
     return df_output
 
 
+@pytest.fixture
+def stdout_args(test_dir):
+    args_dict = {
+        "args": Namespace(
+            output=sys.stdout,
+        )
+    }
+    return args_dict
+
+
+@pytest.fixture
+def output_args(test_dir):
+    path = test_dir / "test_outputs" / "test_outputs_file_io"
+    args_dict = {
+        "args": Namespace(
+            output=path,
+        )
+    }
+    return args_dict
+
+
 # test make_output_directory()
 
 
@@ -476,3 +497,26 @@ def test_writing_df_exists(testing_df, null_logger, making_output_dir):
     file_io.write_out_df(
         testing_df, "test_writing_existing_df", making_output_dir, null_logger, True
     )
+
+
+# test write_out_failed_scrapes
+
+
+def test_write_out_failed_scrapes_stdout(stdout_args, null_logger):
+    """Test write_out_failed_scrapes when args.output is sys.STDOUT"""
+    file_io.write_out_failed_scrapes(
+        ["url_1 - message", "url_2 - fail reason"],
+        "time_stamp",
+        stdout_args["args"],
+        null_logger,
+    )
+
+
+def test_write_out_failed_scrapes(output_args, null_logger):
+    """Test write_out_failed_scrapes when args.output is sys.STDOUT"""
+    file_io.write_out_failed_scrapes(
+        ["url_1 - message", "url_2 - fail reason"],
+        "time_stamp",
+        output_args["args"],
+        null_logger,
+    )
diff --git a/tests/test_inputs/test_inputs_webscraper/cazy_classpage.html b/tests/test_inputs/test_inputs_crawler/cazy_classpage.html
similarity index 100%
rename from tests/test_inputs/test_inputs_webscraper/cazy_classpage.html
rename to tests/test_inputs/test_inputs_crawler/cazy_classpage.html
diff --git a/tests/test_inputs/test_inputs_webscraper/cazy_classpage_no_subfams.html b/tests/test_inputs/test_inputs_crawler/cazy_classpage_no_subfams.html
similarity index 100%
rename from tests/test_inputs/test_inputs_webscraper/cazy_classpage_no_subfams.html
rename to tests/test_inputs/test_inputs_crawler/cazy_classpage_no_subfams.html
diff --git a/tests/test_inputs/test_inputs_webscraper/cazy_gh147_page.html b/tests/test_inputs/test_inputs_crawler/cazy_gh147_page.html
similarity index 100%
rename from tests/test_inputs/test_inputs_webscraper/cazy_gh147_page.html
rename to tests/test_inputs/test_inputs_crawler/cazy_gh147_page.html
diff --git a/tests/test_inputs/test_inputs_webscraper/cazy_gh1_page.html b/tests/test_inputs/test_inputs_crawler/cazy_gh1_page.html
similarity index 100%
rename from tests/test_inputs/test_inputs_webscraper/cazy_gh1_page.html
rename to tests/test_inputs/test_inputs_crawler/cazy_gh1_page.html
diff --git a/tests/test_inputs/test_inputs_webscraper/cazy_homepage.html b/tests/test_inputs/test_inputs_crawler/cazy_homepage.html
similarity index 100%
rename from tests/test_inputs/test_inputs_webscraper/cazy_homepage.html
rename to tests/test_inputs/test_inputs_crawler/cazy_homepage.html
diff --git a/tests/test_inputs/test_inputs_webscraper/cazy_homepage_no_spip_out.html b/tests/test_inputs/test_inputs_crawler/cazy_homepage_no_spip_out.html
similarity index 100%
rename from tests/test_inputs/test_inputs_webscraper/cazy_homepage_no_spip_out.html
rename to tests/test_inputs/test_inputs_crawler/cazy_homepage_no_spip_out.html
diff --git a/tests/test_inputs/test_inputs_crawler/cazy_no_links.html b/tests/test_inputs/test_inputs_crawler/cazy_no_links.html
new file mode 100644
index 00000000..6ce13b09
--- /dev/null
+++ b/tests/test_inputs/test_inputs_crawler/cazy_no_links.html
@@ -0,0 +1,528 @@
+
+
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="fr" lang="fr" dir="ltr">
+<head>
+<title>CAZy - GH147</title> 
+<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+
+
+<meta name="generator" content="SPIP 3.0.1 [19436]" />
+
+
+<link rel="alternate" type="application/rss+xml" title="Syndiquer tout le site" href="spip.php?page=backend" />
+
+
+<link rel="stylesheet" href="squelettes/css/section.css" type="text/css" media="projection, screen, tv" />
+
+
+<link rel="stylesheet" href="squelettes/css/main.css" type="text/css" media="projection, screen, tv" />
+
+
+<link rel="shortcut icon" type="image/x-icon" href="squelettes/images/cazy.ico" />
+
+
+<script type="text/javascript" src="squelettes/js/canevas.js"></script />
+
+
+
+
+<script src="prive/javascript/jquery.js" type="text/javascript"></script>
+
+<script src="prive/javascript/jquery.form.js" type="text/javascript"></script>
+
+<script src="prive/javascript/jquery.autosave.js" type="text/javascript"></script>
+
+<script src="prive/javascript/jquery.placeholder-label.js" type="text/javascript"></script>
+
+<script src="prive/javascript/ajaxCallback.js" type="text/javascript"></script>
+
+<script src="prive/javascript/jquery.cookie.js" type="text/javascript"></script>
+<script type="text/javascript">/*<![CDATA[*/
+var _gaq = _gaq || [];
+_gaq.push(["_setAccount", "UA-12604774-1"]);
+_gaq.push(["_trackPageview"]);
+(function() {
+	var ga = document.createElement("script"); ga.type = "text/javascript"; ga.async = true;
+	ga.src = ("https:" == document.location.protocol ? "https://ssl" : "http://www") + ".google-analytics.com/ga.js";
+	var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(ga, s);
+})();
+/*]]>*/</script>
+<script type="text/javascript">/* <![CDATA[ */
+var box_settings = {tt_img:true,sel_g:"#documents_portfolio a[type='image/jpeg'],#documents_portfolio a[type='image/png'],#documents_portfolio a[type='image/gif']",sel_c:".mediabox",trans:"elastic",speed:"200",ssSpeed:"2500",maxW:"90%",maxH:"90%",minW:"400px",minH:"",opa:"0.9",str_ssStart:"Diaporama",str_ssStop:"Arrêter",str_cur:"{current}/{total}",str_prev:"Précédent",str_next:"Suivant",str_close:"Fermer",splash_url:""};
+/* ]]> */</script>
+<!-- insert_head_css --><link rel="stylesheet" href="plugins-dist/mediabox/colorbox/black-striped/colorbox.css" type="text/css" media="all" /><!-- insert_head -->
+	<script src="plugins-dist/mediabox/javascript/jquery.colorbox.js?1337524663" type="text/javascript"></script>
+	<script src="plugins-dist/mediabox/javascript/spip.mediabox.js?1337524663" type="text/javascript"></script><link rel='stylesheet' type='text/css' media='all' href='plugins-dist/porte_plume/css/barre_outils.css' />
+<link rel='stylesheet' type='text/css' media='all' href='spip.php?page=barre_outils_icones.css' />
+<script type='text/javascript' src='plugins-dist/porte_plume/javascript/jquery.markitup_pour_spip.js'></script>
+<script type='text/javascript' src='plugins-dist/porte_plume/javascript/jquery.previsu_spip.js'></script>
+<script type='text/javascript' src='spip.php?page=porte_plume_start.js&amp;lang=fr'></script>
+
+<!-- CS vide -->
+
+
+<!-- Debut CS -->
+<script type="text/javascript">
+var cs_prive=window.location.pathname.match(/\/ecrire\/$/)!=null;
+jQuery.fn.cs_todo=function(){return this.not('.cs_done').addClass('cs_done');};
+var cs_init = function() {
+	if(window.location.search.match(/page=forum/)!=null) jQuery("legend:contains('Vorgeschaltete Moderation')", this).next().html('Ce forum est modéré à priori : votre contribution n\'apparaîtra qu\'après avoir été validée par un administrateur du site, sauf si vous êtes identifié et autorisé à poster directement.');
+}
+if(typeof onAjaxLoad=='function') onAjaxLoad(cs_init);
+if(window.jQuery) {
+var cs_sel_jQuery=typeof jQuery(document).selector=='undefined'?'@':'';
+var cs_CookiePlugin="prive/javascript/jquery.cookie.js";
+jQuery(document).ready(function(){
+	cs_init.apply(document);
+});
+}
+// --></script>
+
+<!-- Fin CS --></head>
+
+
+ 
+ 
+ 
+ 
+ 
+ 
+
+
+<body onLoad="encours('ezcl','GH147')" class="bkg">
+
+
+
+
+	<!-- cadre_haut -->
+<div class="cadre_haut"> 
+
+<div class="cadre_menu"><!-- cadre_menu -->
+
+<div class="cadre_sousmenu"><!-- cadre_sousmenu -->
+
+<!-- sousmenu1 -->
+<div onmouseout="MouseOutDeclencheSablier()" onmouseover="MouseOverArretSablier()" id="home" class="SousMenuLine">
+&nbsp;
+</div>
+
+</div><!-- cadre_sousmenu --> 
+
+
+</div><!-- cadre_haut -->
+	<!-- cadre_principal -->
+	<div class="cadre_principal">
+		
+		<!-- volet_central -->
+		<div class="volet_central">
+
+
+<div align="right">
+<div style="position:absolute;right:460px;">
+<form name="nextpage">
+<select name="cazypage" onchange="document.location.href='' + this.options[this.selectedIndex].value + ''">
+<option value="http://www.cazy.org/GH1.html" 
+>GH1</option>
+<option value="http://www.cazy.org/GH2.html" 
+>GH2</option>
+<option value="http://www.cazy.org/GH3.html" 
+>GH3</option>
+<option value="http://www.cazy.org/GH4.html" 
+>GH4</option>
+<option value="http://www.cazy.org/GH5.html" 
+>GH5</option>
+<option value="http://www.cazy.org/GH6.html" 
+>GH6</option>
+<option value="http://www.cazy.org/GH7.html" 
+>GH7</option>
+<option value="http://www.cazy.org/GH8.html" 
+>GH8</option>
+<option value="http://www.cazy.org/GH9.html" 
+>GH9</option>
+<option value="http://www.cazy.org/GH10.html" 
+>GH10</option>
+<option value="http://www.cazy.org/GH11.html" 
+>GH11</option>
+<option value="http://www.cazy.org/GH12.html" 
+>GH12</option>
+<option value="http://www.cazy.org/GH13.html" 
+>GH13</option>
+<option value="http://www.cazy.org/GH14.html" 
+>GH14</option>
+<option value="http://www.cazy.org/GH15.html" 
+>GH15</option>
+<option value="http://www.cazy.org/GH16.html" 
+>GH16</option>
+<option value="http://www.cazy.org/GH17.html" 
+>GH17</option>
+<option value="http://www.cazy.org/GH18.html" 
+>GH18</option>
+<option value="http://www.cazy.org/GH19.html" 
+>GH19</option>
+<option value="http://www.cazy.org/GH20.html" 
+>GH20</option>
+<option value="http://www.cazy.org/GH21.html" 
+>GH21</option>
+<option value="http://www.cazy.org/GH22.html" 
+>GH22</option>
+<option value="http://www.cazy.org/GH23.html" 
+>GH23</option>
+<option value="http://www.cazy.org/GH24.html" 
+>GH24</option>
+<option value="http://www.cazy.org/GH25.html" 
+>GH25</option>
+<option value="http://www.cazy.org/GH26.html" 
+>GH26</option>
+<option value="http://www.cazy.org/GH27.html" 
+>GH27</option>
+<option value="http://www.cazy.org/GH28.html" 
+>GH28</option>
+<option value="http://www.cazy.org/GH29.html" 
+>GH29</option>
+<option value="http://www.cazy.org/GH30.html" 
+>GH30</option>
+<option value="http://www.cazy.org/GH31.html" 
+>GH31</option>
+<option value="http://www.cazy.org/GH32.html" 
+>GH32</option>
+<option value="http://www.cazy.org/GH33.html" 
+>GH33</option>
+<option value="http://www.cazy.org/GH34.html" 
+>GH34</option>
+<option value="http://www.cazy.org/GH35.html" 
+>GH35</option>
+<option value="http://www.cazy.org/GH36.html" 
+>GH36</option>
+<option value="http://www.cazy.org/GH37.html" 
+>GH37</option>
+<option value="http://www.cazy.org/GH38.html" 
+>GH38</option>
+<option value="http://www.cazy.org/GH39.html" 
+>GH39</option>
+<option value="http://www.cazy.org/GH40.html" 
+>GH40</option>
+<option value="http://www.cazy.org/GH41.html" 
+>GH41</option>
+<option value="http://www.cazy.org/GH42.html" 
+>GH42</option>
+<option value="http://www.cazy.org/GH43.html" 
+>GH43</option>
+<option value="http://www.cazy.org/GH44.html" 
+>GH44</option>
+<option value="http://www.cazy.org/GH45.html" 
+>GH45</option>
+<option value="http://www.cazy.org/GH46.html" 
+>GH46</option>
+<option value="http://www.cazy.org/GH47.html" 
+>GH47</option>
+<option value="http://www.cazy.org/GH48.html" 
+>GH48</option>
+<option value="http://www.cazy.org/GH49.html" 
+>GH49</option>
+<option value="http://www.cazy.org/GH50.html" 
+>GH50</option>
+<option value="http://www.cazy.org/GH51.html" 
+>GH51</option>
+<option value="http://www.cazy.org/GH52.html" 
+>GH52</option>
+<option value="http://www.cazy.org/GH53.html" 
+>GH53</option>
+<option value="http://www.cazy.org/GH54.html" 
+>GH54</option>
+<option value="http://www.cazy.org/GH55.html" 
+>GH55</option>
+<option value="http://www.cazy.org/GH56.html" 
+>GH56</option>
+<option value="http://www.cazy.org/GH57.html" 
+>GH57</option>
+<option value="http://www.cazy.org/GH58.html" 
+>GH58</option>
+<option value="http://www.cazy.org/GH59.html" 
+>GH59</option>
+<option value="http://www.cazy.org/GH60.html" 
+>GH60</option>
+<option value="http://www.cazy.org/GH61.html" 
+>GH61</option>
+<option value="http://www.cazy.org/GH62.html" 
+>GH62</option>
+<option value="http://www.cazy.org/GH63.html" 
+>GH63</option>
+<option value="http://www.cazy.org/GH64.html" 
+>GH64</option>
+<option value="http://www.cazy.org/GH65.html" 
+>GH65</option>
+<option value="http://www.cazy.org/GH66.html" 
+>GH66</option>
+<option value="http://www.cazy.org/GH67.html" 
+>GH67</option>
+<option value="http://www.cazy.org/GH68.html" 
+>GH68</option>
+<option value="http://www.cazy.org/GH69.html" 
+>GH69</option>
+<option value="http://www.cazy.org/GH70.html" 
+>GH70</option>
+<option value="http://www.cazy.org/GH71.html" 
+>GH71</option>
+<option value="http://www.cazy.org/GH72.html" 
+>GH72</option>
+<option value="http://www.cazy.org/GH73.html" 
+>GH73</option>
+<option value="http://www.cazy.org/GH74.html" 
+>GH74</option>
+<option value="http://www.cazy.org/GH75.html" 
+>GH75</option>
+<option value="http://www.cazy.org/GH76.html" 
+>GH76</option>
+<option value="http://www.cazy.org/GH77.html" 
+>GH77</option>
+<option value="http://www.cazy.org/GH78.html" 
+>GH78</option>
+<option value="http://www.cazy.org/GH79.html" 
+>GH79</option>
+<option value="http://www.cazy.org/GH80.html" 
+>GH80</option>
+<option value="http://www.cazy.org/GH81.html" 
+>GH81</option>
+<option value="http://www.cazy.org/GH82.html" 
+>GH82</option>
+<option value="http://www.cazy.org/GH83.html" 
+>GH83</option>
+<option value="http://www.cazy.org/GH84.html" 
+>GH84</option>
+<option value="http://www.cazy.org/GH85.html" 
+>GH85</option>
+<option value="http://www.cazy.org/GH86.html" 
+>GH86</option>
+<option value="http://www.cazy.org/GH87.html" 
+>GH87</option>
+<option value="http://www.cazy.org/GH88.html" 
+>GH88</option>
+<option value="http://www.cazy.org/GH89.html" 
+>GH89</option>
+<option value="http://www.cazy.org/GH90.html" 
+>GH90</option>
+<option value="http://www.cazy.org/GH91.html" 
+>GH91</option>
+<option value="http://www.cazy.org/GH92.html" 
+>GH92</option>
+<option value="http://www.cazy.org/GH93.html" 
+>GH93</option>
+<option value="http://www.cazy.org/GH94.html" 
+>GH94</option>
+<option value="http://www.cazy.org/GH95.html" 
+>GH95</option>
+<option value="http://www.cazy.org/GH96.html" 
+>GH96</option>
+<option value="http://www.cazy.org/GH97.html" 
+>GH97</option>
+<option value="http://www.cazy.org/GH98.html" 
+>GH98</option>
+<option value="http://www.cazy.org/GH99.html" 
+>GH99</option>
+<option value="http://www.cazy.org/GH100.html" 
+>GH100</option>
+<option value="http://www.cazy.org/GH101.html" 
+>GH101</option>
+<option value="http://www.cazy.org/GH102.html" 
+>GH102</option>
+<option value="http://www.cazy.org/GH103.html" 
+>GH103</option>
+<option value="http://www.cazy.org/GH104.html" 
+>GH104</option>
+<option value="http://www.cazy.org/GH105.html" 
+>GH105</option>
+<option value="http://www.cazy.org/GH106.html" 
+>GH106</option>
+<option value="http://www.cazy.org/GH107.html" 
+>GH107</option>
+<option value="http://www.cazy.org/GH108.html" 
+>GH108</option>
+<option value="http://www.cazy.org/GH109.html" 
+>GH109</option>
+<option value="http://www.cazy.org/GH110.html" 
+>GH110</option>
+<option value="http://www.cazy.org/GH111.html" 
+>GH111</option>
+<option value="http://www.cazy.org/GH112.html" 
+>GH112</option>
+<option value="http://www.cazy.org/GH113.html" 
+>GH113</option>
+<option value="http://www.cazy.org/GH114.html" 
+>GH114</option>
+<option value="http://www.cazy.org/GH115.html" 
+>GH115</option>
+<option value="http://www.cazy.org/GH116.html" 
+>GH116</option>
+<option value="http://www.cazy.org/GH117.html" 
+>GH117</option>
+<option value="http://www.cazy.org/GH118.html" 
+>GH118</option>
+<option value="http://www.cazy.org/GH119.html" 
+>GH119</option>
+<option value="http://www.cazy.org/GH120.html" 
+>GH120</option>
+<option value="http://www.cazy.org/GH121.html" 
+>GH121</option>
+<option value="http://www.cazy.org/GH122.html" 
+>GH122</option>
+<option value="http://www.cazy.org/GH123.html" 
+>GH123</option>
+<option value="http://www.cazy.org/GH124.html" 
+>GH124</option>
+<option value="http://www.cazy.org/GH125.html" 
+>GH125</option>
+<option value="http://www.cazy.org/GH126.html" 
+>GH126</option>
+<option value="http://www.cazy.org/GH127.html" 
+>GH127</option>
+<option value="http://www.cazy.org/GH128.html" 
+>GH128</option>
+<option value="http://www.cazy.org/GH129.html" 
+>GH129</option>
+<option value="http://www.cazy.org/GH130.html" 
+>GH130</option>
+<option value="http://www.cazy.org/GH131.html" 
+>GH131</option>
+<option value="http://www.cazy.org/GH132.html" 
+>GH132</option>
+<option value="http://www.cazy.org/GH133.html" 
+>GH133</option>
+<option value="http://www.cazy.org/GH134.html" 
+>GH134</option>
+<option value="http://www.cazy.org/GH135.html" 
+>GH135</option>
+<option value="http://www.cazy.org/GH136.html" 
+>GH136</option>
+<option value="http://www.cazy.org/GH137.html" 
+>GH137</option>
+<option value="http://www.cazy.org/GH138.html" 
+>GH138</option>
+<option value="http://www.cazy.org/GH139.html" 
+>GH139</option>
+<option value="http://www.cazy.org/GH140.html" 
+>GH140</option>
+<option value="http://www.cazy.org/GH141.html" 
+>GH141</option>
+<option value="http://www.cazy.org/GH142.html" 
+>GH142</option>
+<option value="http://www.cazy.org/GH143.html" 
+>GH143</option>
+<option value="http://www.cazy.org/GH144.html" 
+>GH144</option>
+<option value="http://www.cazy.org/GH145.html" 
+>GH145</option>
+<option value="http://www.cazy.org/GH146.html" 
+>GH146</option>
+<option value="http://www.cazy.org/GH147.html" 
+ selected>GH147</option>
+<option value="http://www.cazy.org/GH148.html" 
+>GH148</option>
+<option value="http://www.cazy.org/GH149.html" 
+>GH149</option>
+<option value="http://www.cazy.org/GH150.html" 
+>GH150</option>
+<option value="http://www.cazy.org/GH151.html" 
+>GH151</option>
+<option value="http://www.cazy.org/GH152.html" 
+>GH152</option>
+<option value="http://www.cazy.org/GH153.html" 
+>GH153</option>
+<option value="http://www.cazy.org/GH154.html" 
+>GH154</option>
+<option value="http://www.cazy.org/GH155.html" 
+>GH155</option>
+<option value="http://www.cazy.org/GH156.html" 
+>GH156</option>
+<option value="http://www.cazy.org/GH157.html" 
+>GH157</option>
+<option value="http://www.cazy.org/GH158.html" 
+>GH158</option>
+<option value="http://www.cazy.org/GH159.html" 
+>GH159</option>
+<option value="http://www.cazy.org/GH160.html" 
+>GH160</option>
+<option value="http://www.cazy.org/GH161.html" 
+>GH161</option>
+<option value="http://www.cazy.org/GH162.html" 
+>GH162</option>
+<option value="http://www.cazy.org/GH163.html" 
+>GH163</option>
+<option value="http://www.cazy.org/GH164.html" 
+>GH164</option>
+<option value="http://www.cazy.org/GH165.html" 
+>GH165</option>
+<option value="http://www.cazy.org/GH166.html" 
+>GH166</option>
+<option value="http://www.cazy.org/GH167.html" 
+>GH167</option>
+<option value="http://www.cazy.org/GH168.html" 
+>GH168</option>
+<option value="http://www.cazy.org/GH0.html" 
+>NC</option>
+ 
+</select><input onclick="gothere()" value="Go" type="button" class="btn_deroul">
+</form> 
+</div>
+
+
+<div style="position:absolute;right:50px;"><a name="formulaire_recherche" id="formulaire_recherche"></a>
+<form name="searchform" action="search" method="get" ENCTYPE="x-www-form-urlencoded">
+        <input name="page" value="recherche" type="hidden"
+/>
+        
+        <input type="text" class="forml" name="recherche" id="q" value="..." onfocus="this.value='';" />
+	 <select name="tag" id="tag">	
+	  <option value="11" >Family</option>	
+	  <option value="1" >Tax_id</option>
+	  <option value="2" >Organism name</option>
+	  <option value="3" >Protein name</option>	  
+	  <option value="4" >GenBank accession</option>
+	  <option value="10" >Uniprot accession</option>	  
+	  <option value="8" >PDB accession</option>
+	  <option value="12" >Carb_ligand</option>
+	  <option value="9" >EC number</option>	 	 	  
+	  <option value="5" >Known activities</option>
+	  <option value="6" >Mechanisms</option>
+	  <option value="7" >Clan</option>	
+	  <option value="">Site</option>
+	</select>
+	<input type="submit" class="bouton" value="Go"/>
+</form>
+
+
+
+
+
+</div>
+</div>
+
+<div><h1 style="color:#00688B;">Glycoside Hydrolase Family 147 
+</h1></div>
+
+
+<div class="afficher_par">
+ <div class="pos_sum">
+    </div>
+
+ 
+</div>
+
+	<br><br>
+	Last update: <b>2020-11-08</b>
+
+&copy; Copyright <b>1998-2020</b><br>
+</div>	
+</div>	
+<!-- cadre_principal --> <script type="text/javascript">
+var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
+document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
+</script> <script type="text/javascript">
+try {
+var pageTracker = _gat._getTracker("UA-12604774-1");
+pageTracker._trackPageview();
+} catch(err) {}</script>
+</body>
+</html>
+
diff --git a/tests/test_inputs/test_inputs_webscraper/protein_no_genbank.html b/tests/test_inputs/test_inputs_crawler/protein_no_genbank.html
similarity index 100%
rename from tests/test_inputs/test_inputs_webscraper/protein_no_genbank.html
rename to tests/test_inputs/test_inputs_crawler/protein_no_genbank.html
diff --git a/tests/test_inputs/test_inputs_webscraper/protein_with_ec.html b/tests/test_inputs/test_inputs_crawler/protein_with_ec.html
similarity index 100%
rename from tests/test_inputs/test_inputs_webscraper/protein_with_ec.html
rename to tests/test_inputs/test_inputs_crawler/protein_with_ec.html
diff --git a/tests/test_inputs/test_inputs_webscraper/protein_with_genbank_synonyms.html b/tests/test_inputs/test_inputs_crawler/protein_with_genbank_synonyms.html
similarity index 100%
rename from tests/test_inputs/test_inputs_webscraper/protein_with_genbank_synonyms.html
rename to tests/test_inputs/test_inputs_crawler/protein_with_genbank_synonyms.html
diff --git a/tests/test_inputs/test_inputs_webscraper/protein_without_ec.html b/tests/test_inputs/test_inputs_crawler/protein_without_ec.html
similarity index 100%
rename from tests/test_inputs/test_inputs_webscraper/protein_without_ec.html
rename to tests/test_inputs/test_inputs_crawler/protein_without_ec.html
diff --git a/tests/test_inputs/test_inputs_webscraper/subfamily_urls.txt b/tests/test_inputs/test_inputs_crawler/subfamily_urls.txt
similarity index 100%
rename from tests/test_inputs/test_inputs_webscraper/subfamily_urls.txt
rename to tests/test_inputs/test_inputs_crawler/subfamily_urls.txt
diff --git a/tests/test_inputs/test_inputs_crawler/test_family_urls.txt b/tests/test_inputs/test_inputs_crawler/test_family_urls.txt
new file mode 100644
index 00000000..4920cce4
--- /dev/null
+++ b/tests/test_inputs/test_inputs_crawler/test_family_urls.txt
@@ -0,0 +1,169 @@
+http://www.cazy.org/GH1.html
+http://www.cazy.org/GH2.html
+http://www.cazy.org/GH3.html
+http://www.cazy.org/GH4.html
+http://www.cazy.org/GH5.html
+http://www.cazy.org/GH6.html
+http://www.cazy.org/GH7.html
+http://www.cazy.org/GH8.html
+http://www.cazy.org/GH9.html
+http://www.cazy.org/GH10.html
+http://www.cazy.org/GH11.html
+http://www.cazy.org/GH12.html
+http://www.cazy.org/GH13.html
+http://www.cazy.org/GH14.html
+http://www.cazy.org/GH15.html
+http://www.cazy.org/GH16.html
+http://www.cazy.org/GH17.html
+http://www.cazy.org/GH18.html
+http://www.cazy.org/GH19.html
+http://www.cazy.org/GH20.html
+http://www.cazy.org/GH21.html
+http://www.cazy.org/GH22.html
+http://www.cazy.org/GH23.html
+http://www.cazy.org/GH24.html
+http://www.cazy.org/GH25.html
+http://www.cazy.org/GH26.html
+http://www.cazy.org/GH27.html
+http://www.cazy.org/GH28.html
+http://www.cazy.org/GH29.html
+http://www.cazy.org/GH30.html
+http://www.cazy.org/GH31.html
+http://www.cazy.org/GH32.html
+http://www.cazy.org/GH33.html
+http://www.cazy.org/GH34.html
+http://www.cazy.org/GH35.html
+http://www.cazy.org/GH36.html
+http://www.cazy.org/GH37.html
+http://www.cazy.org/GH38.html
+http://www.cazy.org/GH39.html
+http://www.cazy.org/GH40.html
+http://www.cazy.org/GH41.html
+http://www.cazy.org/GH42.html
+http://www.cazy.org/GH43.html
+http://www.cazy.org/GH44.html
+http://www.cazy.org/GH45.html
+http://www.cazy.org/GH46.html
+http://www.cazy.org/GH47.html
+http://www.cazy.org/GH48.html
+http://www.cazy.org/GH49.html
+http://www.cazy.org/GH50.html
+http://www.cazy.org/GH51.html
+http://www.cazy.org/GH52.html
+http://www.cazy.org/GH53.html
+http://www.cazy.org/GH54.html
+http://www.cazy.org/GH55.html
+http://www.cazy.org/GH56.html
+http://www.cazy.org/GH57.html
+http://www.cazy.org/GH58.html
+http://www.cazy.org/GH59.html
+http://www.cazy.org/GH60.html
+http://www.cazy.org/GH61.html
+http://www.cazy.org/GH62.html
+http://www.cazy.org/GH63.html
+http://www.cazy.org/GH64.html
+http://www.cazy.org/GH65.html
+http://www.cazy.org/GH66.html
+http://www.cazy.org/GH67.html
+http://www.cazy.org/GH68.html
+http://www.cazy.org/GH69.html
+http://www.cazy.org/GH70.html
+http://www.cazy.org/GH71.html
+http://www.cazy.org/GH72.html
+http://www.cazy.org/GH73.html
+http://www.cazy.org/GH74.html
+http://www.cazy.org/GH75.html
+http://www.cazy.org/GH76.html
+http://www.cazy.org/GH77.html
+http://www.cazy.org/GH78.html
+http://www.cazy.org/GH79.html
+http://www.cazy.org/GH80.html
+http://www.cazy.org/GH81.html
+http://www.cazy.org/GH82.html
+http://www.cazy.org/GH83.html
+http://www.cazy.org/GH84.html
+http://www.cazy.org/GH85.html
+http://www.cazy.org/GH86.html
+http://www.cazy.org/GH87.html
+http://www.cazy.org/GH88.html
+http://www.cazy.org/GH89.html
+http://www.cazy.org/GH90.html
+http://www.cazy.org/GH91.html
+http://www.cazy.org/GH92.html
+http://www.cazy.org/GH93.html
+http://www.cazy.org/GH94.html
+http://www.cazy.org/GH95.html
+http://www.cazy.org/GH96.html
+http://www.cazy.org/GH97.html
+http://www.cazy.org/GH98.html
+http://www.cazy.org/GH99.html
+http://www.cazy.org/GH100.html
+http://www.cazy.org/GH101.html
+http://www.cazy.org/GH102.html
+http://www.cazy.org/GH103.html
+http://www.cazy.org/GH104.html
+http://www.cazy.org/GH105.html
+http://www.cazy.org/GH106.html
+http://www.cazy.org/GH107.html
+http://www.cazy.org/GH108.html
+http://www.cazy.org/GH109.html
+http://www.cazy.org/GH110.html
+http://www.cazy.org/GH111.html
+http://www.cazy.org/GH112.html
+http://www.cazy.org/GH113.html
+http://www.cazy.org/GH114.html
+http://www.cazy.org/GH115.html
+http://www.cazy.org/GH116.html
+http://www.cazy.org/GH117.html
+http://www.cazy.org/GH118.html
+http://www.cazy.org/GH119.html
+http://www.cazy.org/GH120.html
+http://www.cazy.org/GH121.html
+http://www.cazy.org/GH122.html
+http://www.cazy.org/GH123.html
+http://www.cazy.org/GH124.html
+http://www.cazy.org/GH125.html
+http://www.cazy.org/GH126.html
+http://www.cazy.org/GH127.html
+http://www.cazy.org/GH128.html
+http://www.cazy.org/GH129.html
+http://www.cazy.org/GH130.html
+http://www.cazy.org/GH131.html
+http://www.cazy.org/GH132.html
+http://www.cazy.org/GH133.html
+http://www.cazy.org/GH134.html
+http://www.cazy.org/GH135.html
+http://www.cazy.org/GH136.html
+http://www.cazy.org/GH137.html
+http://www.cazy.org/GH138.html
+http://www.cazy.org/GH139.html
+http://www.cazy.org/GH140.html
+http://www.cazy.org/GH141.html
+http://www.cazy.org/GH142.html
+http://www.cazy.org/GH143.html
+http://www.cazy.org/GH144.html
+http://www.cazy.org/GH145.html
+http://www.cazy.org/GH146.html
+http://www.cazy.org/GH147.html
+http://www.cazy.org/GH148.html
+http://www.cazy.org/GH149.html
+http://www.cazy.org/GH150.html
+http://www.cazy.org/GH151.html
+http://www.cazy.org/GH152.html
+http://www.cazy.org/GH153.html
+http://www.cazy.org/GH154.html
+http://www.cazy.org/GH155.html
+http://www.cazy.org/GH156.html
+http://www.cazy.org/GH157.html
+http://www.cazy.org/GH158.html
+http://www.cazy.org/GH159.html
+http://www.cazy.org/GH160.html
+http://www.cazy.org/GH161.html
+http://www.cazy.org/GH162.html
+http://www.cazy.org/GH163.html
+http://www.cazy.org/GH164.html
+http://www.cazy.org/GH165.html
+http://www.cazy.org/GH166.html
+http://www.cazy.org/GH167.html
+http://www.cazy.org/GH168.html
+http://www.cazy.org/GH0.html
diff --git a/tests/test_parse.py b/tests/test_parse.py
index 0dff1487..7b1eb751 100644
--- a/tests/test_parse.py
+++ b/tests/test_parse.py
@@ -37,7 +37,7 @@
 
 from Bio.PDB import PDBList
 
-from scraper import cazy_webscraper, parse
+from scraper import cazy_webscraper, crawler, parse
 
 
 @pytest.fixture
@@ -48,9 +48,9 @@ def out_dir(test_dir):
 
 @pytest.fixture
 def family():
-    family = cazy_webscraper.Family("GH1_test", "GHtest")
+    family = crawler.Family("GH1_test", "GHtest")
 
-    protein = cazy_webscraper.Protein(
+    protein = crawler.Protein(
         "protein_name",
         "GH1",
         "1.2.3.4",
@@ -63,7 +63,7 @@ def family():
     protein = None
     family.members.add(protein)
 
-    protein = cazy_webscraper.Protein(
+    protein = crawler.Protein(
         "protein",
         "GH1",
         "",
@@ -233,7 +233,13 @@ def sequence_fasta(test_dir):
 # test proteins_to_dataframe() (dataframe building function)
 
 
-def test_prt_to_df_ds_fam_no_subfams(args_ds_fam_no_subfam, family, null_logger, monkeypatch):
+def test_prt_to_df_ds_fam_no_subfams(
+    time_stamp,
+    args_ds_fam_no_subfam,
+    family,
+    null_logger,
+    monkeypatch,
+):
     """Test proteins_to_dataframe when data split is family and subfamilies is False."""
 
     with patch("builtins.open", mock_open(read_data="data")) as mock_file:
@@ -245,10 +251,21 @@ def mock_no_return(*args, **kwargs):
         monkeypatch.setattr(parse, "get_structures_and_sequences", mock_no_return)
         monkeypatch.setattr(json, "dump", mock_no_return)
 
-        parse.proteins_to_dataframe([family], args_ds_fam_no_subfam["args"], null_logger)
+        parse.proteins_to_dataframe(
+            [family],
+            time_stamp,
+            args_ds_fam_no_subfam["args"],
+            null_logger,
+        )
 
 
-def test_prt_to_df_ds_class_subfams(args_ds_class_subfam, family, null_logger, monkeypatch):
+def test_prt_to_df_ds_class_subfams(
+    time_stamp,
+    args_ds_class_subfam,
+    family,
+    null_logger,
+    monkeypatch,
+):
     """Test proteins_to_dataframe when data split is class and subfamilies is True."""
 
     with patch("builtins.open", mock_open(read_data="data")) as mock_file:
@@ -259,10 +276,10 @@ def mock_no_return(*args, **kwargs):
         monkeypatch.setattr(parse, "write_out_df", mock_no_return)
         monkeypatch.setattr(parse, "get_genbank_fasta", mock_no_return)
 
-        parse.proteins_to_dataframe([family], args_ds_class_subfam["args"], null_logger)
+        parse.proteins_to_dataframe([family], time_stamp, args_ds_class_subfam["args"], null_logger)
 
 
-def test_prt_to_df_ds_none(args_ds_none, family, null_logger, monkeypatch):
+def test_prt_to_df_ds_none(time_stamp, args_ds_none, family, null_logger, monkeypatch):
     """Test proteins_to_dataframe when data split is None."""
 
     with patch("builtins.open", mock_open(read_data="data")) as mock_file:
@@ -273,7 +290,7 @@ def mock_no_return(*args, **kwargs):
         monkeypatch.setattr(parse, "write_out_df", mock_no_return)
         monkeypatch.setattr(parse, "get_pdb_structures", mock_no_return)
 
-        parse.proteins_to_dataframe([family], args_ds_none["args"], null_logger)
+        parse.proteins_to_dataframe([family], time_stamp, args_ds_none["args"], null_logger)
 
 
 # test get_structures_and_sequences()
diff --git a/tests/test_webscraper.py b/tests/test_webscraper.py
index d31e6f01..f3c9b2b0 100644
--- a/tests/test_webscraper.py
+++ b/tests/test_webscraper.py
@@ -17,24 +17,20 @@
 
 # The MIT License
 
-"""Tests the script cazy_webscraper.py which crawls the CAZy database and retrieves protein data.
+"""Tests the script cazy_webscraper.py which coordinates the scraping of CAZy.
 
 These test are intened to be run from the root of the repository using:
 pytest -v
-
 """
 
 import json
 import pytest
 import sys
-import types
 
 from argparse import Namespace, ArgumentParser
 from requests.exceptions import MissingSchema
 
-from bs4 import BeautifulSoup
-
-from scraper import cazy_webscraper, file_io, parse, utilities
+from scraper import cazy_webscraper, crawler, file_io, parse, utilities
 
 
 @pytest.fixture
@@ -50,7 +46,7 @@ def output_dir(test_dir):
 
 
 @pytest.fixture
-def args_datasplit_none_sub_false():
+def get_cazy_data_args_none():
     argsdict = {
         "args": Namespace(
             data_split=None,
@@ -61,18 +57,7 @@ def args_datasplit_none_sub_false():
 
 
 @pytest.fixture
-def args_datasplit_none_sub_true():
-    argsdict = {
-        "args": Namespace(
-            data_split=None,
-            subfamilies=True,
-        )
-    }
-    return argsdict
-
-
-@pytest.fixture
-def args_datasplit_family():
+def get_cazy_data_args_family():
     argsdict = {
         "args": Namespace(
             data_split="family",
@@ -83,10 +68,11 @@ def args_datasplit_family():
 
 
 @pytest.fixture
-def args_datasplit_class():
+def get_cazy_data_args_class():
     argsdict = {
         "args": Namespace(
-            data_split="class"
+            data_split="class",
+            subfamilies=True,
         )
     }
     return argsdict
@@ -95,145 +81,16 @@ def args_datasplit_class():
 @pytest.fixture
 def config_dict():
     configuration_dict = {
-        "Glycoside Hydrolases (GHs)": ["GH1"]
+        "Glycoside Hydrolases (GHs)": ["GH1"],
+        "Polysaccharide Lyases (PLs)": None,
     }
     return configuration_dict
 
 
 @pytest.fixture
-def args_datasplit_family_sub_false():
-    argsdict = {
-        "args": Namespace(
-            data_split="family",
-            subfamilies=False,
-        )
-    }
-    return argsdict
-
-
-@pytest.fixture
-def cazy_home_page(input_dir):
-    file_path = input_dir / "cazy_homepage.html"
-    return file_path
-
-
-@pytest.fixture
-def cazy_home_no_spip(input_dir):
-    file_path = input_dir / "cazy_homepage_no_spip_out.html"
-    return file_path
-
-
-@pytest.fixture
-def cazy_class_page(input_dir):
-    file_path = input_dir / "cazy_classpage.html"
-    return file_path
-
-
-@pytest.fixture
-def family_urls(input_dir):
-    file_path = input_dir / "family_urls.txt"
-    with open(file_path, "r") as fh:
-        fam_string = fh.read()
-    fam_string = fam_string[1:-1]
-    fam_string = fam_string.replace("'", "")
-    fam_list = fam_string.split(", ")
-    return fam_list
-
-
-@pytest.fixture
-def family_h3_element(cazy_class_page):
-    with open(cazy_class_page) as fp:
-        soup = BeautifulSoup(fp, features="lxml")
-
-    return [_ for _ in
-            soup.find_all("h3", {"class": "spip"}) if
-            str(_.contents[0]) == "Tables for Direct Access"][0]
-
-
-@pytest.fixture
-def subfamily_urls(input_dir):
-    file_path = input_dir / "subfamily_urls.txt"
-    with open(file_path, "r") as fh:
-        fam_string = fh.read()
-    fam_string = fam_string[1:-1]
-    fam_string = fam_string.replace("'", "")
-    fam_list = fam_string.split(", ")
-    return fam_list
-
-
-@pytest.fixture
-def no_subfam_h3_element(input_dir):
-    file_path = input_dir / "cazy_classpage_no_subfams.html"
-    with open(file_path) as fp:
-        soup = BeautifulSoup(fp, features="lxml")
-
-    return [_ for _ in
-            soup.find_all("h3", {"class": "spip"}) if
-            str(_.contents[0]) == "Tables for Direct Access"][0]
-
-
-@pytest.fixture
-def gh1_page(input_dir):
-    file_path = input_dir / "cazy_gh1_page.html"
-    return file_path
-
-
-@pytest.fixture
-def gh147_page(input_dir):
-    file_path = input_dir / "cazy_gh147_page.html"
-    return file_path
-
-
-@pytest.fixture
-def protein_list():
-    return [
-        cazy_webscraper.Protein("protein_name", "GH1", "1.2.3.4", "organism"),
-        cazy_webscraper.Protein(
-            "protein",
-            "GH1",
-            "",
-            "organism",
-            {"GenBank": ["link1"], "UniProt": ["link2"], "PDB": ["link3"]},
-        ),
-    ]
-
-
-@pytest.fixture
-def protein_gen(protein_list):
-    return (_ for _ in protein_list)
-
-
-@pytest.fixture
-def full_protein_gen(input_dir):
-    file_path = input_dir / "proteins.txt"
-    with open(file_path) as fh:
-        proteins = fh.read().splitlines()
-    gen = (x for x in proteins)
-    return gen
-
-
-@pytest.fixture
-def protein_without_ec(input_dir):
-    file_path = input_dir / "protein_without_ec.html"
-    return file_path
-
-
-@pytest.fixture
-def protein_with_ec(input_dir):
-    file_path = input_dir / "protein_with_ec.html"
-    return file_path
-
-
-@pytest.fixture
-def protein_with_gb_synonyms(input_dir):
-    file_path = input_dir / "protein_with_genbank_synonyms.html"
-    return file_path
-
-
-@pytest.fixture
-def protein_with_no_gb(input_dir):
-    file_path = input_dir / "protein_no_genbank.html"
-    return file_path
+def mock_family():
+    family = crawler.Family("GH1", "Glycoside_Hydrolases(GH)")
+    return family
 
 
 # test classes
@@ -241,7 +98,7 @@ def protein_with_no_gb(input_dir):
 
 def test_protein_get_protein_dict():
     """Test the Protein class __str__ and __repr__."""
-    protein = cazy_webscraper.Protein(
+    protein = crawler.Protein(
         "protein_name",
         "GH1",
         ["1.2.3.4"],
@@ -266,7 +123,7 @@ def test_protein_get_protein_dict():
 
 def test_family_get_name():
     """Tests get family name for Family."""
-    family = cazy_webscraper.Family("GH1", "Glycoside_Hydrolases(GH)")
+    family = crawler.Family("GH1", "Glycoside_Hydrolases(GH)")
 
     family_name = family.get_family_name()
 
@@ -303,9 +160,10 @@ def mock_parser(*args, **kwargs):
             force=False,
             nodelete=False,
             genbank="dummy_email",
-            genbank_output= output_dir,
+            genbank_output=output_dir,
             pdb="pdb",
-            pdb_output= output_dir,
+            pdb_output=output_dir,
+            retries=0,
         )
         return parser
 
@@ -359,6 +217,7 @@ def mock_parser(*args, **kwargs):
             genbank_output=output_dir,
             pdb="pdb",
             pdb_output=output_dir,
+            retries=0,
         )
         return parser
 
@@ -387,763 +246,288 @@ def mock_retrieving_cazy_data(*args, **kwargs):
 # test get_cazy_data
 
 
-def test_get_cazy_data_class_urls_none(cazy_home_url, null_logger, monkeypatch):
-    """Test get_cazy_data, checking that when no class page urls are returned the program exits."""
-
-    def mock_failed_class_pages(*args, **kwargs):
-        return
-
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_class_urls", mock_failed_class_pages)
-
-    with pytest.raises(SystemExit) as pytest_wrapped_e:
-        cazy_webscraper.get_cazy_data(cazy_home_url, None, None, None, null_logger, None)
-    assert pytest_wrapped_e.type == SystemExit
-
-
-def test_get_cazy_data_class_urls_empty_list(cazy_home_url, null_logger, monkeypatch):
-    """Test get_cazy_data, checking that when no class page urls are returned the program exits."""
-
-    def mock_failed_class_pages(*args, **kwargs):
-        return []
-
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_class_urls", mock_failed_class_pages)
-
-    with pytest.raises(SystemExit) as pytest_wrapped_e:
-        cazy_webscraper.get_cazy_data(cazy_home_url, None, None, None, null_logger, None)
-    assert pytest_wrapped_e.type == SystemExit
-
-
-def test_get_cazy_data_family_urls_none(
-    cazy_home_url,
-    cazy_dictionary,
-    args_datasplit_none_sub_false,
-    null_logger,
-    monkeypatch,
-):
-    """Test get_cazy_data when nothing returned from get_cazy_family_urls."""
-    with open(cazy_dictionary, "r") as fh:
-        cazy_dict = json.load(fh)
-
-    def mock_class_pages(*args, **kwargs):
-        return ["http://www.cazy.org/Glycoside-Hydrolases.html"]
-
-    def mock_family_urls_none(*args, **kwargs):
-        return
-
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_class_urls", mock_class_pages)
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_family_urls", mock_family_urls_none)
-
-    cazy_webscraper.get_cazy_data(
-        cazy_home_url,
-        None,
-        None,
-        cazy_dict,
-        null_logger,
-        args_datasplit_none_sub_false["args"]
-    )
-
-
-def test_get_cazy_data_no_config_ds_family(
-    cazy_home_url,
-    cazy_dictionary,
-    args_datasplit_family,
-    null_logger,
-    monkeypatch,
-):
-    """Test get_cazy_data wit no config dict and data split is 'family'"""
-    with open(cazy_dictionary, "r") as fh:
-        cazy_dict = json.load(fh)
-
-    def mock_class_pages(*args, **kwargs):
-        return ["http://www.cazy.org/Glycoside-Hydrolases.html"]
-
-    def mock_family_urls(*args, **kwargs):
-        return ["cazy.org/fam", "http://www.cazy.org/GH1.html"]
-
-    def mock_parsing_family(*args, **kwargs):
-        family = Namespace(
-            name="cazy fam",
-            cazy_class="cazy_class",
-            members=set([1, 2, 3])
-        )
-        return family
-
-    def mock_building_dataframe(*args, **kwargs):
-        return
-
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_class_urls", mock_class_pages)
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_family_urls", mock_family_urls)
-    monkeypatch.setattr(cazy_webscraper, "parse_family", mock_parsing_family)
-    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_building_dataframe)
-
-    cazy_webscraper.get_cazy_data(
-        cazy_home_url,
-        None,
-        None,
-        cazy_dict,
-        null_logger,
-        args_datasplit_family["args"]
-    )
-
-
-def test_get_cazy_data_no_config_ds_class(
-    cazy_home_url,
-    cazy_dictionary,
-    args_datasplit_class,
-    null_logger,
-    monkeypatch,
-):
-    """Test get_cazy_data wit no config dict and data split is 'class'"""
-    with open(cazy_dictionary, "r") as fh:
-        cazy_dict = json.load(fh)
-
-    def mock_class_pages(*args, **kwargs):
-        return ["http://www.cazy.org/Glycoside-Hydrolases.html"]
-
-    def mock_family_urls(*args, **kwargs):
-        return ["http://www.cazy.org/GH1.html"]
-
-    def mock_parsing_family(*args, **kwargs):
-        family = Namespace(
-            name="cazy fam",
-            cazy_class="cazy_class",
-            members=set([1, 2, 3])
-        )
-        return family
-
-    def mock_building_dataframe(*args, **kwargs):
-        return
-
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_class_urls", mock_class_pages)
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_family_urls", mock_family_urls)
-    monkeypatch.setattr(cazy_webscraper, "parse_family", mock_parsing_family)
-    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_building_dataframe)
-
-    cazy_webscraper.get_cazy_data(
-        cazy_home_url,
-        None,
-        None,
-        cazy_dict,
-        null_logger,
-        args_datasplit_class["args"]
-    )
-
-
-def test_get_cazy_data_no_config_ds_class_lenfamilies_0(
-    cazy_home_url,
-    cazy_dictionary,
-    args_datasplit_class,
-    null_logger,
-    monkeypatch,
-):
-    """Test get_cazy_data wit no config dict and data split is class, and len(families) == 0"""
-    with open(cazy_dictionary, "r") as fh:
-        cazy_dict = json.load(fh)
-
-    def mock_class_pages(*args, **kwargs):
-        return ["http://www.cazy.org/Glycoside-Hydrolases.html"]
-
-    def mock_family_urls(*args, **kwargs):
-        return []
-
-    def mock_parsing_family(*args, **kwargs):
-        family = Namespace(
-            name="cazy fam",
-            cazy_class="cazy_class",
-            members=set([1, 2, 3])
-        )
-        return family
-
-    def mock_building_dataframe(*args, **kwargs):
-        return
-
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_class_urls", mock_class_pages)
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_family_urls", mock_family_urls)
-    monkeypatch.setattr(cazy_webscraper, "parse_family", mock_parsing_family)
-    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_building_dataframe)
-
-    cazy_webscraper.get_cazy_data(
-        cazy_home_url,
-        None,
-        None,
-        cazy_dict,
-        null_logger,
-        args_datasplit_class["args"]
-    )
-
-
-def test_get_cazy_data_no_config_ds_none_lenfamilies_0(
+def test_get_cazy_data_no_fam_urls(
     cazy_home_url,
     cazy_dictionary,
-    args_datasplit_none_sub_false,
+    config_dict,
+    time_stamp,
     null_logger,
-    monkeypatch,
+    get_cazy_data_args_none,
+    monkeypatch
 ):
-    """Test get_cazy_data wit no config dict and data split is None, and len(families) == 0"""
+    """Test get_cazy_data."""
     with open(cazy_dictionary, "r") as fh:
         cazy_dict = json.load(fh)
 
-    def mock_class_pages(*args, **kwargs):
-        return ["http://www.cazy.org/Glycoside-Hydrolases.html"]
-
-    def mock_family_urls(*args, **kwargs):
-        return []
-
-    def mock_parsing_family(*args, **kwargs):
-        family = Namespace(
-            name="cazy fam",
-            cazy_class="cazy_class",
-            members=set([1, 2, 3])
-        )
-        return family
+    def mock_get_class_urls(*args, **kwargs):
+        return [
+            ["http://www.cazy.org/Glycoside-Hydrolases.html", 0],
+            ["http://www.cazy.org/Polysaccharide-Lyases.html", 0],
+        ]
 
-    def mock_building_dataframe(*args, **kwargs):
+    def mock_no_return(*args, **kwargs):
         return
 
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_class_urls", mock_class_pages)
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_family_urls", mock_family_urls)
-    monkeypatch.setattr(cazy_webscraper, "parse_family", mock_parsing_family)
-    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_building_dataframe)
+    monkeypatch.setattr(cazy_webscraper, "get_class_urls", mock_get_class_urls)
+    monkeypatch.setattr(crawler, "get_cazy_family_urls", mock_no_return)
+    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_no_return)
+    monkeypatch.setattr(file_io, "write_out_failed_scrapes", mock_no_return)
 
     cazy_webscraper.get_cazy_data(
         cazy_home_url,
         None,
-        None,
+        config_dict,
         cazy_dict,
+        1,
+        time_stamp,
         null_logger,
-        args_datasplit_none_sub_false["args"]
+        get_cazy_data_args_none["args"],
     )
 
 
-def test_get_cazy_data_no_config_ds_none(
+def test_get_cazy_data_no_fam_returned(
+    time_stamp,
     cazy_home_url,
     cazy_dictionary,
-    args_datasplit_none_sub_false,
+    config_dict,
     null_logger,
-    monkeypatch,
+    get_cazy_data_args_none,
+    monkeypatch
 ):
-    """Test get_cazy_data wit no config dict and data split is 'None'"""
+    """Test get_cazy_data() when data_split is None, subfamilies is False."""
     with open(cazy_dictionary, "r") as fh:
         cazy_dict = json.load(fh)
 
-    def mock_class_pages(*args, **kwargs):
-        return ["http://www.cazy.org/Glycoside-Hydrolases.html"]
+    def mock_get_class_urls(*args, **kwargs):
+        return [
+            ["http://www.cazy.org/Glycoside-Hydrolases.html", 0],
+            ["http://www.cazy.org/Polysaccharide-Lyases.html", 0],
+        ]
 
-    def mock_family_urls(*args, **kwargs):
-        return ["http://www.cazy.org/GH1.html"]
+    def mock_get_family_urls(*args, **kwargs):
+        return [
+            "dddwww.cazddddy.org/GH1.html",
+            "http://www.cazy.org/GH1.html",
+            "http://www.cazy.org/PL1.html",
+        ]
 
-    def mock_parsing_family(*args, **kwargs):
-        family = Namespace(
-            name="cazy fam",
-            cazy_class="cazy_class",
-            members=set([1, 2, 3])
-        )
-        return family
+    def mock_parse_family(*args, **kwargs):
+        return [None, "mock error message"]
 
-    def mock_building_dataframe(*args, **kwargs):
+    def mock_no_return(*args, **kwargs):
         return
 
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_class_urls", mock_class_pages)
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_family_urls", mock_family_urls)
-    monkeypatch.setattr(cazy_webscraper, "parse_family", mock_parsing_family)
-    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_building_dataframe)
+    monkeypatch.setattr(cazy_webscraper, "get_class_urls", mock_get_class_urls)
+    monkeypatch.setattr(crawler, "get_cazy_family_urls", mock_get_family_urls)
+    monkeypatch.setattr(crawler, "parse_family", mock_parse_family)
+    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_no_return)
+    monkeypatch.setattr(file_io, "write_out_failed_scrapes", mock_no_return)
 
     cazy_webscraper.get_cazy_data(
         cazy_home_url,
         None,
-        None,
+        config_dict,
         cazy_dict,
+        1,
+        time_stamp,
         null_logger,
-        args_datasplit_none_sub_false["args"]
+        get_cazy_data_args_none["args"],
     )
 
 
-def test_get_cazy_data_config_ds_fam(
+def test_get_cazy_data_subfams_split_fam(
+    time_stamp,
     cazy_home_url,
     cazy_dictionary,
     config_dict,
-    args_datasplit_family,
+    mock_family,
     null_logger,
-    monkeypatch,
+    get_cazy_data_args_family,
+    monkeypatch
 ):
-    """Test get_cazy_data with a config dict and data split is 'family'"""
+    """Test get_cazy_data() when data split is family and subfamily is True."""
     with open(cazy_dictionary, "r") as fh:
         cazy_dict = json.load(fh)
 
-    def mock_class_pages(*args, **kwargs):
-        return ["http://www.cazy.org/Glycoside-Hydrolases.html"]
+    def mock_get_class_urls(*args, **kwargs):
+        return [
+            ["http://www.cazy.org/Glycoside-Hydrolases.html", 0],
+            ["http://www.cazy.org/Polysaccharide-Lyases.html", 0],
+        ]
 
-    def mock_family_urls(*args, **kwargs):
-        return ["cazy.fam.org", "http://www.cazy.org/GH1.html"]
+    def mock_get_family_urls(*args, **kwargs):
+        return [
+            "dddwww.cazddddy.org/GH1.html",
+            "http://www.cazy.org/GH1.html",
+            "http://www.cazy.org/PL1.html",
+        ]
 
-    def mock_parsing_family(*args, **kwargs):
-        family = Namespace(
-            name="cazy fam",
-            cazy_class="cazy_class",
-            members=set([1, 2, 3])
-        )
-        return family
+    def mock_parse_family(*args, **kwargs):
+        return [mock_family, None]
 
-    def mock_building_dataframe(*args, **kwargs):
+    def mock_no_return(*args, **kwargs):
         return
 
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_class_urls", mock_class_pages)
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_family_urls", mock_family_urls)
-    monkeypatch.setattr(cazy_webscraper, "parse_family", mock_parsing_family)
-    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_building_dataframe)
+    monkeypatch.setattr(cazy_webscraper, "get_class_urls", mock_get_class_urls)
+    monkeypatch.setattr(crawler, "get_cazy_family_urls", mock_get_family_urls)
+    monkeypatch.setattr(crawler, "parse_family", mock_parse_family)
+    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_no_return)
+    monkeypatch.setattr(file_io, "write_out_failed_scrapes", mock_no_return)
 
     cazy_webscraper.get_cazy_data(
         cazy_home_url,
         None,
         config_dict,
         cazy_dict,
+        1,
+        time_stamp,
         null_logger,
-        args_datasplit_family["args"]
+        get_cazy_data_args_family["args"],
     )
 
 
-def test_get_cazy_data_ds_none_subfams_false(
+def test_get_cazy_data_subfams_split_class(
+    time_stamp,
     cazy_home_url,
     cazy_dictionary,
     config_dict,
-    args_datasplit_none_sub_false,
+    mock_family,
     null_logger,
-    monkeypatch,
+    get_cazy_data_args_class,
+    monkeypatch
 ):
-    """Test get_cazy_data with a config dict and data split is 'None' and subfamilies is False"""
+    """Test get_cazy_data() when data split is class and subfamily is True."""
     with open(cazy_dictionary, "r") as fh:
         cazy_dict = json.load(fh)
 
-    def mock_class_pages(*args, **kwargs):
-        return ["http://www.cazy.org/Glycoside-Hydrolases.html"]
+    def mock_get_class_urls(*args, **kwargs):
+        return [
+            ["http://www.cazy.org/Glycoside-Hydrolases.html", 0],
+            ["http://www.cazy.org/Polysaccharide-Lyases.html", 0],
+        ]
 
-    def mock_family_urls(*args, **kwargs):
-        return ["http://www.cazy.org/GH1.html"]
+    def mock_get_family_urls(*args, **kwargs):
+        return [
+            "dddwww.cazddddy.org/GH1.html",
+            "http://www.cazy.org/GH1.html",
+            "http://www.cazy.org/PL1.html",
+        ]
 
-    def mock_parsing_family(*args, **kwargs):
-        family = Namespace(
-            name="cazy fam",
-            cazy_class="cazy_class",
-            members=set([1, 2, 3])
-        )
-        return family
+    def mock_parse_family(*args, **kwargs):
+        return [mock_family, None]
 
-    def mock_building_dataframe(*args, **kwargs):
+    def mock_no_return(*args, **kwargs):
         return
 
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_class_urls", mock_class_pages)
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_family_urls", mock_family_urls)
-    monkeypatch.setattr(cazy_webscraper, "parse_family", mock_parsing_family)
-    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_building_dataframe)
+    monkeypatch.setattr(cazy_webscraper, "get_class_urls", mock_get_class_urls)
+    monkeypatch.setattr(crawler, "get_cazy_family_urls", mock_get_family_urls)
+    monkeypatch.setattr(crawler, "parse_family", mock_parse_family)
+    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_no_return)
+    monkeypatch.setattr(file_io, "write_out_failed_scrapes", mock_no_return)
 
     cazy_webscraper.get_cazy_data(
         cazy_home_url,
         None,
         config_dict,
         cazy_dict,
+        1,
+        time_stamp,
         null_logger,
-        args_datasplit_none_sub_false["args"],
+        get_cazy_data_args_class["args"],
     )
 
 
-def test_get_cazy_data_ds_none_subfams_true(
+def test_get_cazy_data_split_none(
+    time_stamp,
     cazy_home_url,
     cazy_dictionary,
     config_dict,
-    args_datasplit_none_sub_true,
+    mock_family,
     null_logger,
-    monkeypatch,
+    get_cazy_data_args_none,
+    monkeypatch
 ):
-    """Test get_cazy_data with a config dict and data split is 'None' and subfamilies is True"""
+    """Test get_cazy_data() when data split is None and subfamily is False."""
     with open(cazy_dictionary, "r") as fh:
         cazy_dict = json.load(fh)
 
-    def mock_class_pages(*args, **kwargs):
-        return ["http://www.cazy.org/Glycoside-Hydrolases.html"]
+    def mock_get_class_urls(*args, **kwargs):
+        return [
+            ["http://www.cazy.org/Glycoside-Hydrolases.html", 0],
+            ["http://www.cazy.org/Polysaccharide-Lyases.html", 0],
+        ]
 
-    def mock_family_urls(*args, **kwargs):
-        return ["http://www.cazy.org/GH5_3.html"]
+    def mock_get_family_urls(*args, **kwargs):
+        return [
+            "dddwww.cazddddy.org/GH1.html",
+            "http://www.cazy.org/GH1.html",
+            "http://www.cazy.org/PL1.html",
+        ]
 
-    def mock_parsing_family(*args, **kwargs):
-        family = Namespace(
-            name="cazy fam",
-            cazy_class="cazy_class",
-            members=set([1, 2, 3])
-        )
-        return family
+    def mock_parse_family(*args, **kwargs):
+        return [mock_family, None]
 
-    def mock_building_dataframe(*args, **kwargs):
+    def mock_no_return(*args, **kwargs):
         return
 
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_class_urls", mock_class_pages)
-    monkeypatch.setattr(cazy_webscraper, "get_cazy_family_urls", mock_family_urls)
-    monkeypatch.setattr(cazy_webscraper, "parse_family", mock_parsing_family)
-    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_building_dataframe)
+    monkeypatch.setattr(cazy_webscraper, "get_class_urls", mock_get_class_urls)
+    monkeypatch.setattr(crawler, "get_cazy_family_urls", mock_get_family_urls)
+    monkeypatch.setattr(crawler, "parse_family", mock_parse_family)
+    monkeypatch.setattr(parse, "proteins_to_dataframe", mock_no_return)
+    monkeypatch.setattr(file_io, "write_out_failed_scrapes", mock_no_return)
 
     cazy_webscraper.get_cazy_data(
         cazy_home_url,
         None,
         config_dict,
         cazy_dict,
+        1,
+        time_stamp,
         null_logger,
-        args_datasplit_none_sub_true["args"],
+        get_cazy_data_args_none["args"],
     )
 
 
-# test get_cazy_class_urls()
-
-
-def test_get_class_urls_exclusions_none(cazy_home_url, cazy_home_page, null_logger, monkeypatch):
-    """Test get_cazy_class_urls when excluded_classess is None."""
-    with open(cazy_home_page, "r") as fp:
-        home_page = BeautifulSoup(fp, features="lxml")
-
-        def mock_get_home_page(*args, **kwargs):
-            return [home_page, None]
-
-        monkeypatch.setattr(cazy_webscraper, "get_page", mock_get_home_page)
-
-        expected_result = [
-            'http://www.cazy.org/Glycoside-Hydrolases.html',
-            'http://www.cazy.org/GlycosylTransferases.html',
-            'http://www.cazy.org/Polysaccharide-Lyases.html',
-            'http://www.cazy.org/Carbohydrate-Esterases.html',
-            'http://www.cazy.org/Auxiliary-Activities.html',
-            'http://www.cazy.org/Carbohydrate-Binding-Modules.html',
-            ]
-
-        assert expected_result == cazy_webscraper.get_cazy_class_urls(
-            cazy_home_url,
-            None,
-            null_logger,
-        )
+# test get_class_urls
 
 
-def test_get_class_urls_fail(cazy_home_url, null_logger, monkeypatch):
-    """Test get_cazy_class_urls home_page not returned"""
+def test_get_class_urls_no_urls(cazy_home_url, null_logger, monkeypatch):
+    """Test get_class_urls() when an empty list is returned."""
 
-    def mock_get_home_page(*args, **kwargs):
-        return [None, "error"]
+    def mock_class_urls(*args, **kwargs):
+        return []
 
-    monkeypatch.setattr(cazy_webscraper, "get_page", mock_get_home_page)
+    monkeypatch.setattr(crawler, "get_cazy_class_urls", mock_class_urls)
 
     with pytest.raises(SystemExit) as pytest_wrapped_e:
-        cazy_webscraper.get_cazy_class_urls(cazy_home_url, None, null_logger)
+        cazy_webscraper.get_class_urls(cazy_home_url, None, 1, null_logger)
     assert pytest_wrapped_e.type == SystemExit
 
 
-def test_get_class_urls_exclusions_given(
-    cazy_home_url,
-    cazy_home_page,
-    null_logger,
-    monkeypatch,
-):
-    """Test get_cazy_class_urls when excluded_classess is not None."""
-    with open(cazy_home_page) as fp:
-        soup = BeautifulSoup(fp, features="lxml")
-
-    exclusions = ["<strong>Glycoside Hydrolases (GHs)</strong>"]
-
-    def mock_get_home_page(*args, **kwargs):
-        return [soup, None]
+def test_get_class_urls_none_urls(cazy_home_url, null_logger, monkeypatch):
+    """Test get_class_urls() when a None type object is returned."""
 
-    monkeypatch.setattr(cazy_webscraper, "get_page", mock_get_home_page)
+    def mock_class_urls(*args, **kwargs):
+        return
 
-    expected_result = [
-        'http://www.cazy.org/GlycosylTransferases.html',
-        'http://www.cazy.org/Polysaccharide-Lyases.html',
-        'http://www.cazy.org/Carbohydrate-Esterases.html',
-        'http://www.cazy.org/Auxiliary-Activities.html',
-        'http://www.cazy.org/Carbohydrate-Binding-Modules.html',
-        ]
-
-    assert expected_result == cazy_webscraper.get_cazy_class_urls(
-        cazy_home_url,
-        exclusions,
-        null_logger,
-    )
+    monkeypatch.setattr(crawler, "get_cazy_class_urls", mock_class_urls)
 
+    with pytest.raises(SystemExit) as pytest_wrapped_e:
+        cazy_webscraper.get_class_urls(cazy_home_url, None, 1, null_logger)
+    assert pytest_wrapped_e.type == SystemExit
 
-def test_get_class_urls_attribute(
-    cazy_home_url,
-    cazy_home_no_spip,
-    null_logger,
-    monkeypatch,
-):
-    """Test get_cazy_class_urls when attribute error is raised."""
-    with open(cazy_home_no_spip) as fp:
-        soup = fp.read()
 
-    exclusions = ["<strong>Glycoside Hydrolases (GHs)</strong>"]
+def test_get_class_urls_success(cazy_home_url, null_logger, monkeypatch):
+    """Test get_class_urls() when URLs were returned."""
 
-    def mock_get_home_page(*args, **kwargs):
-        return [soup, None]
+    def mock_class_urls(*args, **kwargs):
+        return ["url1", "url2", "url3"]
 
-    monkeypatch.setattr(cazy_webscraper, "get_page", mock_get_home_page)
+    monkeypatch.setattr(crawler, "get_cazy_class_urls", mock_class_urls)
 
-    assert None is cazy_webscraper.get_cazy_class_urls(
+    assert [['url1', 0], ['url2', 0], ['url3', 0]] == cazy_webscraper.get_class_urls(
         cazy_home_url,
-        exclusions,
-        null_logger,
-    )
-
-
-# test get_cazy_family_urls
-
-
-def test_get_family_urls_fail(args_datasplit_family, null_logger, monkeypatch):
-    """Test get_cazy_family_urls when no page is returned."""
-
-    def mock_get_page(*args, **kwargs):
-        return [None, "error"]
-
-    monkeypatch.setattr(cazy_webscraper, "get_page", mock_get_page)
-
-    assert None is cazy_webscraper.get_cazy_family_urls(
-        "class_url",
-        "cazy_home_url",
-        "class_name",
-        args_datasplit_family["args"],
-        null_logger,
-    )
-
-
-def test_get_family_urls_success(
-    cazy_class_page,
-    args_datasplit_family,
-    family_urls,
-    null_logger,
-    monkeypatch,
-):
-    """Test get_cazy_family_urls when successful, and subfamilies is True."""
-    with open(cazy_class_page) as fp:
-        page = BeautifulSoup(fp, features="lxml")
-
-    def mock_get_page(*args, **kwargs):
-        return [page, None]
-
-    def mock_get_subfams(*args, **kwargs):
-        return []
-
-    monkeypatch.setattr(cazy_webscraper, "get_page", mock_get_page)
-    monkeypatch.setattr(cazy_webscraper, "get_subfamily_links", mock_get_subfams)
-
-    assert family_urls == cazy_webscraper.get_cazy_family_urls(
-        "class_url",
-        "http://www.cazy.org",
-        "class_name",
-        args_datasplit_family["args"],
-        null_logger,
-    )
-
-
-# test get_subfamily_links
-
-
-def test_get_subfam_links_len_0(family_h3_element, subfamily_urls, null_logger):
-    """Test get_subfamily_links when no links are retrieved."""
-
-    assert subfamily_urls == cazy_webscraper.get_subfamily_links(
-        family_h3_element,
-        "http://www.cazy.org",
-        null_logger,
-    )
-
-
-def test_get_subfam_links_urls(no_subfam_h3_element, null_logger):
-    """Test get_subfamily_links when urls are retrieved."""
-
-    assert None is cazy_webscraper.get_subfamily_links(
-        no_subfam_h3_element,
-        "http://www.cazy.org",
-        null_logger,
-    )
-
-
-# test parse_family()
-
-
-def test_parse_family(protein_list, null_logger, monkeypatch):
-    """Tests parse_family()"""
-
-    def mock_parsing_family_pages(*args, **kwargs):
-        return protein_list
-
-    monkeypatch.setattr(cazy_webscraper, "parse_family_pages", mock_parsing_family_pages)
-
-    cazy_webscraper.parse_family(
-        "http://www.cazy.org/GH1.html",
-        "GH1",
-        "http://www.cazy.org",
-        null_logger,
-    )
-
-    cazy_webscraper.parse_family(
-        ".cazy.org/GH1.html",
-        "GH1",
-        "http://www.cazy.org",
-        null_logger,
-    )
-
-
-# test parse_family_pages()
-
-
-def test_parse_fam_pages_none(null_logger, monkeypatch):
-    """Test parse_family_pages when no page is returned."""
-    # assert is None
-    def mock_get_page(*args, **kwargs):
-        return [None, "error"]
-
-    monkeypatch.setattr(cazy_webscraper, "get_page", mock_get_page)
-
-    assert [] == cazy_webscraper.parse_family_pages(
-        "http://www.cazy.org/GH1.html",
-        "GH1",
-        "http://www.cazy.org",
+        None,
+        1,
         null_logger,
     )
-
-
-def test_parse_fam_pages_single(gh147_page, protein_gen, null_logger, monkeypatch):
-    """Test parse_family_pages when only one page, raises index error."""
-    with open(gh147_page) as fp:
-        soup = BeautifulSoup(fp, features="lxml")
-
-    def mock_get_page(*args, **kwargs):
-        return [soup, None]
-
-    def mock_parse_proteins(*args, **kwargs):
-        return protein_gen
-
-    monkeypatch.setattr(cazy_webscraper, "get_page", mock_get_page)
-    monkeypatch.setattr(cazy_webscraper, "parse_proteins", mock_parse_proteins)
-
-    assert True is isinstance(
-        (
-            cazy_webscraper.parse_family_pages(
-                "http://www.cazy.org/GH1.html",
-                "GH1",
-                "http://www.cazy.org",
-                null_logger,
-            )
-        ),
-        types.GeneratorType,
-    )
-
-
-def test_parse_fam_pages_multiple(gh1_page, protein_gen, null_logger, monkeypatch):
-    """Test parse_family_pages when there are multiple pages."""
-    with open(gh1_page) as fp:
-        soup = BeautifulSoup(fp, features="lxml")
-
-    def mock_get_page(*args, **kwargs):
-        return [soup, None]
-
-    def mock_parse_proteins(*args, **kwargs):
-        return protein_gen
-
-    monkeypatch.setattr(cazy_webscraper, "get_page", mock_get_page)
-    monkeypatch.setattr(cazy_webscraper, "parse_proteins", mock_parse_proteins)
-
-    assert True is isinstance(
-        (
-            cazy_webscraper.parse_family_pages(
-                "http://www.cazy.org/GH1.html",
-                "GH1",
-                "http://www.cazy.org",
-                null_logger,
-            )
-        ),
-        types.GeneratorType,
-    )
-
-
-# test parse_proteins()
-
-
-def test_parse_proteins_none(null_logger, monkeypatch):
-    """Test parse_proteins when no page is returned."""
-
-    def mock_get_page(*args, **kwargs):
-        return [None, "error"]
-
-    monkeypatch.setattr(cazy_webscraper, "get_page", mock_get_page)
-
-    assert True is isinstance(
-        cazy_webscraper.parse_proteins("protein_url", "family", null_logger),
-        types.GeneratorType,
-    )
-
-
-def test_parse_proteins(gh147_page, null_logger, monkeypatch):
-    """Tests parse_proteins when successful."""
-    with open(gh147_page) as fp:
-        soup = BeautifulSoup(fp, features="lxml")
-
-    def mock_get_page(*args, **kwargs):
-        return soup
-
-    monkeypatch.setattr(cazy_webscraper, "get_page", mock_get_page)
-
-    assert True is isinstance(
-        cazy_webscraper.parse_proteins("protein_url", "family", null_logger),
-        types.GeneratorType,
-    )
-
-
-# test row_to_protein
-
-
-def test_row_to_protein_no_ecs(protein_without_ec):
-    """Test row_to_protein when no EC#s are listed."""
-    with open(protein_without_ec) as fp:
-        row = BeautifulSoup(fp, features="lxml")
-
-    assert True is isinstance(
-        cazy_webscraper.row_to_protein(row, "GH147"),
-        cazy_webscraper.Protein
-    )
-
-
-def test_row_to_protein_ec(protein_with_ec):
-    """Test row_to_protein when EC#s are listed."""
-    with open(protein_with_ec) as fp:
-        row = BeautifulSoup(fp, features="lxml")
-
-    assert True is isinstance(
-        cazy_webscraper.row_to_protein(row, "GH147"),
-        cazy_webscraper.Protein
-    )
-
-
-def test_row_to_protein_gb_synoymns(protein_with_gb_synonyms):
-    """Test row_to_protein when protein has GenBank synonyms."""
-    with open(protein_with_gb_synonyms) as fp:
-        row = BeautifulSoup(fp, features="lxml")
-
-    assert True is isinstance(
-        cazy_webscraper.row_to_protein(row, "GH147"),
-        cazy_webscraper.Protein
-    )
-
-
-def test_row_to_protein_no_gb(protein_with_no_gb):
-    """Test row_to_protein when protein has no GenBank accession."""
-    with open(protein_with_no_gb) as fp:
-        row = BeautifulSoup(fp, features="lxml")
-
-    assert True is isinstance(
-        cazy_webscraper.row_to_protein(row, "GH147"),
-        cazy_webscraper.Protein
-    )
-
-
-# test browser_decorator
-
-
-def test_browser_decorator():
-    """Test browser_decorator to ensure proper handling if unsuccessful."""
-
-    result = cazy_webscraper.get_page('www.caz!!!!!!!!y.org')
-    assert True == (result[0] is None) and (type(result[1]) is MissingSchema)