Repository: omicsNLP/Auto-CORPus
Pull request #149: Refactored config file loading logic for easier default config usage.
Author: login: Thomas-Rowlands
Created: 2025-03-18T14:17:24Z | Merged: 2025-03-25T11:25:31Z

# Description

Config files are now loaded separately depending on whether a default config is used or a new, user-made custom one.

Fixes #143 

This new way of loading a default config file can be done like this:
```
from autocorpus.Autocorpus import Autocorpus
from autocorpus.configs.default_config import DefaultConfig

ac = Autocorpus(base_dir="my_base_dir", main_text="path/to/html/file")
ac.config = DefaultConfig.load_config(DefaultConfig.LEGACY_PMC)
ac.process_files("output/path")

```

For custom config files, users can do the following:
```
from autocorpus.Autocorpus import Autocorpus

ac = Autocorpus(base_dir="my_base_dir", main_text="path/to/html/file")
ac.config = ac.read_config("path/to/my_config.json")
ac.process_files("output/path")

```


## Type of change

- [ ] Documentation (non-breaking change that adds or improves the documentation)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Optimization (non-breaking, back-end change that speeds up the code)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Breaking change (whatever its nature)

## Key checklist

- [ ] All tests pass (eg. `pytest`)
- [ ] The documentation builds and looks OK (eg. `mkdocs`)
- [ ] Pre-commit hooks run successfully (eg. `pre-commit run --all-files`)

## Further checks

- [ ] Code is commented, particularly in hard-to-understand areas
- [ ] Tests added or an issue has been opened to tackle that in the future. (Indicate issue here: # (issue))


Linked issue #143: No way to use bundled config files
While the default config files are now bundled in the wheel, users who install AC from PyPI will still not be able to use them in any kind of reasonable way. When they install it (e.g. with `pipx`), they'll be able to use the `auto-corpus` command *but* all the files associated with the package, including the config files, will be squirrelled away in their filesystem somewhere. I don't think it's a solution just to document this, because the folder is likely to be somewhere obscure -- it's not really intended that users access these sorts of folders directly.

We need some way for users to use the builtin configs via the command-line interface. For example, in addition to the existing `-c`/`--config` flags, we could also have `-b`/`--builtin` flags, used like this:

```sh
auto-corpus -b pmc -t "output" -f "path/to/html/file" -o JSON
```

Internally, AC would look for a file in `wherever/the/package/is/configs` called `{builtin_name}_config.json` then just run as normal after that. (This is just an example but you get the idea.)

It would also be handy to have a command for users to get the list of builtin configs (preferably each should have a short description too), so they know what their options are.

Diff:
diff --git a/autocorpus/Autocorpus.py b/autocorpus/Autocorpus.py
index f7998960..65afb6e2 100644
--- a/autocorpus/Autocorpus.py
+++ b/autocorpus/Autocorpus.py
@@ -1,6 +1,5 @@
 """Auto-CORPus primary functions are called from this script, after initialisation with __main__.py."""
 
-import argparse
 import json
 from pathlib import Path
 
@@ -11,16 +10,28 @@
 from .bioc_formatter import BiocFormatter
 from .section import Section
 from .table import Table
-from .tableimage import TableImage
 from .utils import handle_not_tables
 
 
 class Autocorpus:
     """Parent class for all Auto-CORPus functionality."""
 
-    def __read_config(self, config_path):
-        config_path = Path(config_path)
-        with config_path.open("r") as f:
+    @staticmethod
+    def read_config(config_path: str) -> dict:
+        """Reads a configuration file and returns its content.
+
+        Args:
+            config_path (str): The path to the configuration file.
+
+        Returns:
+            dict: The content of the configuration file.
+
+        Raises:
+            FileNotFoundError: If the configuration file does not exist.
+            json.JSONDecodeError: If the configuration file is not a valid JSON.
+            KeyError: If the configuration file does not contain the expected "config" key.
+        """
+        with open(config_path, encoding="utf-8") as f:
             ## TODO: validate config file here if possible
             content = json.load(f)
             return content["config"]
@@ -345,9 +356,46 @@ def __merge_table_data(self):
         else:
             return
 
+    def process_files(self):
+        """Processes the files specified in the configuration.
+
+        This method performs the following steps:
+        1. Checks if a valid configuration is loaded. If not, raises a RuntimeError.
+        2. Handles the main text file:
+            - Parses the HTML content of the file.
+            - Extracts the main text from the parsed HTML.
+            - Attempts to extract abbreviations from the main text and HTML content.
+              If an error occurs during this process, it prints the error.
+        3. Processes linked tables, if any:
+            - Parses the HTML content of each linked table file.
+        4. Merges table data.
+        5. Checks if there are any documents in the tables and sets the `has_tables` attribute accordingly.
+
+        Raises:
+            RuntimeError: If no valid configuration is loaded.
+        """
+        if not self.config:
+            raise RuntimeError("A valid config file must be loaded.")
+        # handle main_text
+        if self.file_path:
+            soup = self.__handle_html(self.file_path, self.config)
+            self.main_text = self.__extract_text(soup, self.config)
+            try:
+                self.abbreviations = Abbreviations(
+                    self.main_text, soup, self.config, self.file_path
+                ).to_dict()
+            except Exception as e:
+                print(e)
+        if self.linked_tables:
+            for table_file in self.linked_tables:
+                soup = self.__handle_html(table_file, self.config)
+        self.__merge_table_data()
+        if "documents" in self.tables and not self.tables["documents"] == []:
+            self.has_tables = True
+
     def __init__(
         self,
-        config_path,
+        config,
         base_dir=None,
         main_text=None,
         linked_tables=None,
@@ -358,45 +406,27 @@ def __init__(
         """Utilises the input config file to create valid BioC versions of input HTML journal articles.
 
         Args:
-            config_path (str): path to the config file to be used
+            config (dict): configuration file for the input HTML journal articles
             base_dir (str): base directory of the input HTML journal articles
             main_text (str): path to the main text of the article (HTML files only)
             linked_tables (list): list of linked table file paths to be included in this run (HTML files only)
             table_images (list): list of table image file paths to be included in this run (JPEG or PNG files only)
             associated_data_path (str): currently unused
-            trained_data (str): currently unused, previously added for image processing
+            trained_data (list): currently unused
         """
-        # handle common
-        config = self.__read_config(config_path)
         self.base_dir = base_dir
         self.file_path = main_text
+        self.linked_tables = linked_tables
+        self.table_images = table_images
+        self.associated_data_path = associated_data_path
+        self.config = config
+        self.trained_data = trained_data
         self.main_text = {}
         self.empty_tables = {}
         self.tables = {}
         self.abbreviations = {}
         self.has_tables = False
 
-        # handle main_text
-        if self.file_path:
-            soup = self.__handle_html(self.file_path, config)
-            self.main_text = self.__extract_text(soup, config)
-            try:
-                self.abbreviations = Abbreviations(
-                    self.main_text, soup, config, self.file_path
-                ).to_dict()
-            except Exception as e:
-                print(e)
-        if linked_tables:
-            for table_file in linked_tables:
-                soup = self.__handle_html(table_file, config)
-        if table_images:
-            self.tables = TableImage(
-                table_images, self.base_dir, trained_data=trained_data
-            ).to_dict()
-        self.__merge_table_data()
-        if "documents" in self.tables and not self.tables["documents"] == []:
-            self.has_tables = True
-
     def to_bioc(self):
         """Get the currently loaded bioc as a dict.
 
@@ -469,28 +499,3 @@ def to_dict(self):
             "abbreviations": self.abbreviations,
             "tables": self.tables,
         }
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser()
-    parser.add_argument(
-        "-f", "--filepath", type=str, help="filepath of html file to be processed"
-    )
-    parser.add_argument(
-        "-t", "--target_dir", type=str, help="target directory for output"
-    )
-    parser.add_argument(
-        "-c", "--config", type=str, help="filepath for configuration JSON file"
-    )
-    parser.add_argument(
-        "-d", "--config_dir", type=str, help="directory of configuration JSON files"
-    )
-    parser.add_argument(
-        "-a", "--associated_data", type=str, help="directory of associated data"
-    )
-    args = parser.parse_args()
-    filepath = args.filepath
-    target_dir = args.target_dir
-    config_path = args.config
-
-    Autocorpus(config_path, filepath, target_dir).to_file(target_dir)
diff --git a/autocorpus/__main__.py b/autocorpus/__main__.py
index e7ebdf07..4e7a45aa 100644
--- a/autocorpus/__main__.py
+++ b/autocorpus/__main__.py
@@ -9,6 +9,7 @@
 from tqdm import tqdm
 
 from autocorpus.Autocorpus import Autocorpus
+from autocorpus.configs.default_config import DefaultConfig
 
 parser = argparse.ArgumentParser(prog="PROG")
 parser.add_argument(
@@ -46,6 +47,9 @@
 group.add_argument(
     "-d", "--config_dir", type=str, help="directory of configuration JSON files"
 )
+group.add_argument(
+    "-b", "--default_config", type=str, help="name of a default config file"
+)
 
 
 def get_file_type(file_path: Path) -> str:
@@ -160,7 +164,7 @@ def main():
     args = parser.parse_args()
     file_path = Path(args.filepath)
     target_dir = Path(args.target_dir if args.target_dir else "autoCORPus_output")
-    config = args.config
+    config = args.config if args.config else args.default_config
     config_dir = args.config_dir  # noqa: F841 ## TODO: Use this variable
     associated_data = args.associated_data  # noqa: F841 ## TODO: Use this variable
     output_format = args.output_format if args.output_format else "JSON"
@@ -203,15 +207,27 @@ def main():
             )
             base_dir = file_path.parent if not file_path.is_dir() else file_path
             try:
+                if args.config:
+                    config = Autocorpus.read_config(args.config)
+                elif args.default_config:
+                    try:
+                        config = DefaultConfig[args.default_config].load_config()
+                    except KeyError:
+                        raise ValueError(
+                            f"{args.default_config} is not a valid default config."
+                        )
+
                 ac = Autocorpus(
-                    config,
                     base_dir=str(base_dir),
                     main_text=structure[key]["main_text"],
+                    config=config,
                     linked_tables=sorted(structure[key]["linked_tables"]),
                     table_images=sorted(structure[key]["table_images"]),
                     trained_data=trained_data,
                 )
 
+                ac.process_files()
+
                 out_dir = Path(structure[key]["out_dir"])
                 if structure[key]["main_text"]:
                     key = key.replace("\\", "/")
diff --git a/autocorpus/configs/default_config.py b/autocorpus/configs/default_config.py
new file mode 100644
index 00000000..3f073c35
--- /dev/null
+++ b/autocorpus/configs/default_config.py
@@ -0,0 +1,44 @@
+"""Contains a loader for default configuration files."""
+
+import importlib.resources as resources
+import json
+from enum import Enum
+
+
+class DefaultConfig(Enum):
+    """An enumeration representing different configuration files for various datasets.
+
+    Attributes:
+            LEGACY_PMC (str): Configuration file for legacy PMC data (pre-October 2024).
+            PMC (str): Configuration file for current PMC data.
+            PLOS_GENETICS (str): Configuration file for PLOS Genetics data.
+            NATURE_GENETICS (str): Configuration file for Nature Genetics data.
+
+    Methods:
+            load_config():
+                Loads and returns the configuration from the associated file.
+                The configuration is lazy-loaded and cached upon first access.
+
+    """
+
+    LEGACY_PMC = "config_pmc_pre_oct_2024.json"
+    PMC = "config_pmc.json"
+    PLOS_GENETICS = "config_plos_genetics.json"
+    NATURE_GENETICS = "config_nature_genetics.json"
+
+    def __init__(self, filename):
+        """Initializes the DefaultConfig enum with the given filename.
+
+        Args:
+            filename (str): The name of the configuration file to load.
+        """
+        self._filename = filename
+        self._config = None  # Lazy-loaded cache
+
+    def load_config(self):
+        """Loads the configuration file when first accessed."""
+        if self._config is None:
+            config_path = resources.files("autocorpus.configs") / self._filename
+            with config_path.open("r", encoding="utf-8") as f_in:
+                self._config = json.load(f_in)["config"]
+        return self._config
diff --git a/tests/test_regression.py b/tests/test_regression.py
index 17231ad1..0d7089aa 100644
--- a/tests/test_regression.py
+++ b/tests/test_regression.py
@@ -3,6 +3,8 @@
 import json
 from pathlib import Path
 
+from autocorpus.configs.default_config import DefaultConfig
+
 
 def test_autocorpus():
     """A regression test for the main autoCORPus class on the AutoCORPus Paper.
@@ -43,11 +45,13 @@ def test_autocorpus():
         expected_tables = json.load(f)
 
     auto_corpus = Autocorpus(
-        "autocorpus/configs/config_pmc_pre_oct_2024.json",
+        config=DefaultConfig.LEGACY_PMC.load_config(),
         base_dir="tests/data/PMC/Pre-Oct-2024",
         main_text="tests/data/PMC/Pre-Oct-2024/PMC8885717.html",
     )
 
+    auto_corpus.process_files()
+
     abbreviations = auto_corpus.abbreviations
     bioc = auto_corpus.to_bioc()
     tables = auto_corpus.tables


Reviews:
user: login: alexdewar
body: Looks good! There's a missing call to `process_files()` needed and I'd replace that `assert` statement, but other than that I think it's all ok. I've made some other suggestions, but I'll leave it up to you whether you think they're worth it or not.
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-03-18T16:48:51Z
user: login: AdrianDAlessandro
body: Overall pretty good. Alex made some relevant comments that should be addressed
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-18T17:39:37Z
user: login: Thomas-Rowlands
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-19T19:54:53Z
user: login: Thomas-Rowlands
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-19T19:57:28Z
user: login: Thomas-Rowlands
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-19T20:05:14Z
user: login: alexdewar
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-20T10:47:02Z
user: login: Thomas-Rowlands
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-24T13:20:59Z
user: login: Thomas-Rowlands
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-24T13:55:16Z
user: login: Thomas-Rowlands
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-24T13:56:35Z
user: login: AdrianDAlessandro
body: This is really good overall! I've made one suggestion about making the type of `config` consistent, which should also simplify the Enum. Will be happy to merge after that change
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-24T18:00:06Z
user: login: alexdewar
body: Agree with @AdrianDAlessandro's comment, but aside from that it looks good!
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-25T08:45:39Z
user: login: Thomas-Rowlands
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-25T10:25:06Z
user: login: Thomas-Rowlands
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-25T10:25:22Z
user: login: alexdewar
body: LGTM! :rocket: 
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-25T10:39:34Z
user: login: AdrianDAlessandro
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-25T11:15:00Z
user: login: Thomas-Rowlands
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-25T11:15:44Z
user: login: AdrianDAlessandro
body: Great! I've enabled auto-merge and updated the branch, so this will merge once the checks have passed.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-25T11:22:39Z

Review comments:
diff_hunk: @@ -358,9 +350,27 @@ def __merge_table_data(self):
         else:
             return
 
+    def process_files(self):
+        assert self.config, "A valid config file must be loaded."
path: autocorpus/Autocorpus.py
user: login: alexdewar
body: Weirdly in Python if you build a package in optimised mode it strips out `assert` statements, so you can't rely on it for error handling, in case a user has installed it this way.

```suggestion
        if not self.config:
            raise RuntimeError("A valid config file must be loaded.")
```
created_at: 2025-03-18T16:21:01Z
author_association: COLLABORATOR
diff_hunk: @@ -204,14 +208,19 @@ def main():
             base_dir = file_path.parent if not file_path.is_dir() else file_path
             try:
                 ac = Autocorpus(
-                    config,
                     base_dir=str(base_dir),
                     main_text=structure[key]["main_text"],
                     linked_tables=sorted(structure[key]["linked_tables"]),
                     table_images=sorted(structure[key]["table_images"]),
                     trained_data=trained_data,
                 )
-
+                if args.config:
+                    ac.config = ac.read_config(config)
+                elif args.default_config:
+                    default_config = DefaultConfig.string_to_constant(
+                        args.default_config
+                    )
+                    ac.config = DefaultConfig.load_config(default_config)
path: autocorpus/__main__.py
user: login: alexdewar
body: Personally I think it's cleaner to pass the `config` into the `Autocorpus` constructor rather than setting it after the fact. I'm guessing that's why you split out the actual processing to the `process_files()` method?

In any case, you need to call that too, otherwise the input files won't be processed when you run AC from the command line:


```suggestion
                    ac.config = DefaultConfig.load_config(default_config)
                ac.process_files()
```

The regression test didn't catch this because it's exercising a different part of the code, but without it you get errors like this:

```
❯ auto-corpus -c autocorpus/configs/config_pmc_pre_oct_2024.json -t output -f tests/data/PMC/
Pre-Oct-2024/PMC8885717.html
100%|█████| 1/1 [00:00<00:00, 2576.35it/s, file=PMC8885717*, linked_tables=0, table_images=0]
Auto-CORPus has completed processing with some errors. Please inspect the log file for further details.
```
created_at: 2025-03-18T16:41:06Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,56 @@
+import json
+from pathlib import Path
+
+
+class DefaultConfig:
+    """
+    DefaultConfig class provides a way to initialize and load different configurations for the application.
+    Attributes:
+        LEGACY_PMC (int): Identifier for the legacy PMC configuration.
+        PMC (int): Identifier for the PMC configuration.
+        PLOS_GENETICS (int): Identifier for the PLOS Genetics configuration.
+        NATURE_GENETICS (int): Identifier for the Nature Genetics configuration.
+    Methods:
+        string_to_constant(config_name: str) -> int:
+                config (str): The configuration name to load.
+        load_config(config: int) -> dict:
+            Loads the specified configuration file.
+                config (str): The constant id of the default configuration file to load.
+            Returns:
+                dict: The loaded configuration as a dictionary.
+    """
+
+    LEGACY_PMC = 1
+    PMC = 2
+    PLOS_GENETICS = 3
+    NATURE_GENETICS = 4
path: autocorpus/configs/default_config.py
user: login: alexdewar
body: Generally I think using enums where you can makes things cleaner, but in this case I'd be inclined just to accept a string for the default config name then assume the file name is `config_{name}.json`. That way, when you come to add more default configs in future you can just add another file, without having to also update the code.
created_at: 2025-03-18T16:42:52Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,56 @@
+import json
+from pathlib import Path
+
+
+class DefaultConfig:
+    """
+    DefaultConfig class provides a way to initialize and load different configurations for the application.
+    Attributes:
+        LEGACY_PMC (int): Identifier for the legacy PMC configuration.
+        PMC (int): Identifier for the PMC configuration.
+        PLOS_GENETICS (int): Identifier for the PLOS Genetics configuration.
+        NATURE_GENETICS (int): Identifier for the Nature Genetics configuration.
+    Methods:
+        string_to_constant(config_name: str) -> int:
+                config (str): The configuration name to load.
+        load_config(config: int) -> dict:
+            Loads the specified configuration file.
+                config (str): The constant id of the default configuration file to load.
+            Returns:
+                dict: The loaded configuration as a dictionary.
+    """
+
+    LEGACY_PMC = 1
+    PMC = 2
+    PLOS_GENETICS = 3
+    NATURE_GENETICS = 4
+
+    @staticmethod
+    def string_to_constant(config_name: str) -> int:
+        config_name = config_name.upper()
+        if hasattr(DefaultConfig, config_name):
+            return getattr(DefaultConfig, config_name)
+        raise ValueError(
+            f"Invalid default config name: {config_name}. Please check documentation for a list of default configs"
+        )
+
+    @staticmethod
+    def load_config(config: int) -> dict:
+        if config == DefaultConfig.LEGACY_PMC:
+            config_file = "config_pmc_pre_oct_2024.json"
+        elif config == DefaultConfig.PMC:
+            config_file = "config_pmc.json"
+        elif config == DefaultConfig.PLOS_GENETICS:
+            config_file = "config_plos_genetics.json"
+        elif config == DefaultConfig.NATURE_GENETICS:
+            config_file = "config_nature_genetics.json"
+        else:
+            raise Exception(
+                "A valid config was not provided. Please provide a valid DefaultConfig setting."
+            )
+
+        config_path = Path(__file__).parent / config_file
path: autocorpus/configs/default_config.py
user: login: alexdewar
body: This will work, but to find files inside a package it's a bit more robust to use `importlib`, like we've done here: https://github.com/omicsNLP/Auto-CORPus/blob/17ccb2eac1ff7f9c090c424931cbd030a3984e09/autocorpus/utils.py#L81
created_at: 2025-03-18T16:44:37Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,56 @@
+import json
+from pathlib import Path
+
+
+class DefaultConfig:
+    """
+    DefaultConfig class provides a way to initialize and load different configurations for the application.
+    Attributes:
+        LEGACY_PMC (int): Identifier for the legacy PMC configuration.
+        PMC (int): Identifier for the PMC configuration.
+        PLOS_GENETICS (int): Identifier for the PLOS Genetics configuration.
+        NATURE_GENETICS (int): Identifier for the Nature Genetics configuration.
+    Methods:
+        string_to_constant(config_name: str) -> int:
+                config (str): The configuration name to load.
+        load_config(config: int) -> dict:
+            Loads the specified configuration file.
+                config (str): The constant id of the default configuration file to load.
+            Returns:
+                dict: The loaded configuration as a dictionary.
+    """
+
+    LEGACY_PMC = 1
+    PMC = 2
+    PLOS_GENETICS = 3
+    NATURE_GENETICS = 4
+
+    @staticmethod
+    def string_to_constant(config_name: str) -> int:
+        config_name = config_name.upper()
+        if hasattr(DefaultConfig, config_name):
+            return getattr(DefaultConfig, config_name)
+        raise ValueError(
+            f"Invalid default config name: {config_name}. Please check documentation for a list of default configs"
+        )
path: autocorpus/configs/default_config.py
user: login: alexdewar
body: Alternatively:

```suggestion
        try:
            return getattr(DefaultConfig, config_name)
        except AttributeError:
            raise ValueError(
                f"Invalid default config name: {config_name}. Please check documentation for a list of default configs"
            )
```
created_at: 2025-03-18T16:48:18Z
author_association: COLLABORATOR
diff_hunk: @@ -371,44 +381,23 @@ def __init__(
         """Utilises the input config file to create valid BioC versions of input HTML journal articles.
 
         Args:
-            config_path (str): path to the config file to be used
             base_dir (str): base directory of the input HTML journal articles
             main_text (str): path to the main text of the article (HTML files only)
             linked_tables (list): list of linked table file paths to be included in this run (HTML files only)
             table_images (list): list of table image file paths to be included in this run (JPEG or PNG files only)
             associated_data_path (str): currently unused
-            trained_data (str): currently unused, previously added for image processing
         """
-        # handle common
-        config = self.__read_config(config_path)
         self.base_dir = base_dir
         self.file_path = main_text
+        self.linked_tables = linked_tables
+        self.table_images = table_images
+        self.associated_data_path = associated_data_path
         self.main_text = {}
         self.empty_tables = {}
         self.tables = {}
         self.abbreviations = {}
         self.has_tables = False
-
-        # handle main_text
-        if self.file_path:
-            soup = self.__handle_html(self.file_path, config)
-            self.main_text = self.__extract_text(soup, config)
-            try:
-                self.abbreviations = Abbreviations(
-                    self.main_text, soup, config, self.file_path
-                ).to_dict()
-            except Exception as e:
-                print(e)
-        if linked_tables:
-            for table_file in linked_tables:
-                soup = self.__handle_html(table_file, config)
-        if table_images:
-            self.tables = TableImage(
-                table_images, self.base_dir, trained_data=trained_data
-            ).to_dict()
path: autocorpus/Autocorpus.py
user: login: AdrianDAlessandro
body: It seems like this `if table_images` block is missing in the `process_files` method. Is that intentional?
created_at: 2025-03-18T17:15:57Z
author_association: COLLABORATOR
diff_hunk: @@ -204,14 +208,19 @@ def main():
             base_dir = file_path.parent if not file_path.is_dir() else file_path
             try:
                 ac = Autocorpus(
-                    config,
                     base_dir=str(base_dir),
                     main_text=structure[key]["main_text"],
                     linked_tables=sorted(structure[key]["linked_tables"]),
                     table_images=sorted(structure[key]["table_images"]),
                     trained_data=trained_data,
                 )
-
+                if args.config:
+                    ac.config = ac.read_config(config)
+                elif args.default_config:
+                    default_config = DefaultConfig.string_to_constant(
+                        args.default_config
+                    )
+                    ac.config = DefaultConfig.load_config(default_config)
path: autocorpus/__main__.py
user: login: AdrianDAlessandro
body: I'd also lean towards building the config and then passing it to the constructor. However I think this is something that will be changed by the upcoming refactor, so it might not be too important.

And yes, it looks like `process_files` isn't being called anywhere, so that needs to be called after `ac.config` is set
created_at: 2025-03-18T17:19:39Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,56 @@
+import json
+from pathlib import Path
+
+
+class DefaultConfig:
+    """
+    DefaultConfig class provides a way to initialize and load different configurations for the application.
+    Attributes:
+        LEGACY_PMC (int): Identifier for the legacy PMC configuration.
+        PMC (int): Identifier for the PMC configuration.
+        PLOS_GENETICS (int): Identifier for the PLOS Genetics configuration.
+        NATURE_GENETICS (int): Identifier for the Nature Genetics configuration.
+    Methods:
+        string_to_constant(config_name: str) -> int:
+                config (str): The configuration name to load.
+        load_config(config: int) -> dict:
+            Loads the specified configuration file.
+                config (str): The constant id of the default configuration file to load.
+            Returns:
+                dict: The loaded configuration as a dictionary.
+    """
+
+    LEGACY_PMC = 1
+    PMC = 2
+    PLOS_GENETICS = 3
+    NATURE_GENETICS = 4
+
+    @staticmethod
+    def string_to_constant(config_name: str) -> int:
+        config_name = config_name.upper()
+        if hasattr(DefaultConfig, config_name):
+            return getattr(DefaultConfig, config_name)
+        raise ValueError(
+            f"Invalid default config name: {config_name}. Please check documentation for a list of default configs"
+        )
path: autocorpus/configs/default_config.py
user: login: AdrianDAlessandro
body: Also, this seems to be hand-rolling what we get with [Enums](https://docs.python.org/3/howto/enum.html), it might be better to just use them directly.
created_at: 2025-03-18T17:26:46Z
author_association: COLLABORATOR
diff_hunk: @@ -160,7 +164,7 @@ def main():
     args = parser.parse_args()
     file_path = Path(args.filepath)
     target_dir = Path(args.target_dir if args.target_dir else "autoCORPus_output")
-    config = args.config
+    config = args.config if args.config else args.default_config
path: autocorpus/__main__.py
user: login: AdrianDAlessandro
body: Argparse might be smart enough do this automatically, but there should be a check to make sure only one `config` or `default_config` are set, or at least logging a warning to say both have been set and which is being used.
created_at: 2025-03-18T17:32:32Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,56 @@
+import json
+from pathlib import Path
+
+
+class DefaultConfig:
+    """
+    DefaultConfig class provides a way to initialize and load different configurations for the application.
+    Attributes:
+        LEGACY_PMC (int): Identifier for the legacy PMC configuration.
+        PMC (int): Identifier for the PMC configuration.
+        PLOS_GENETICS (int): Identifier for the PLOS Genetics configuration.
+        NATURE_GENETICS (int): Identifier for the Nature Genetics configuration.
+    Methods:
+        string_to_constant(config_name: str) -> int:
+                config (str): The configuration name to load.
+        load_config(config: int) -> dict:
+            Loads the specified configuration file.
+                config (str): The constant id of the default configuration file to load.
+            Returns:
+                dict: The loaded configuration as a dictionary.
+    """
+
+    LEGACY_PMC = 1
+    PMC = 2
+    PLOS_GENETICS = 3
+    NATURE_GENETICS = 4
path: autocorpus/configs/default_config.py
user: login: Thomas-Rowlands
body: The idea here was thinking more along the line of using it as a package rather than focussing on CLI users, but I see your point, the CLI method does use a string input. I think using it as a module, then passing in a string for various provided constants seems odd. Do you mean altering the constants like this? LEGACY_PMC = "config_pmc.json"?
created_at: 2025-03-19T19:54:52Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,56 @@
+import json
+from pathlib import Path
+
+
+class DefaultConfig:
+    """
+    DefaultConfig class provides a way to initialize and load different configurations for the application.
+    Attributes:
+        LEGACY_PMC (int): Identifier for the legacy PMC configuration.
+        PMC (int): Identifier for the PMC configuration.
+        PLOS_GENETICS (int): Identifier for the PLOS Genetics configuration.
+        NATURE_GENETICS (int): Identifier for the Nature Genetics configuration.
+    Methods:
+        string_to_constant(config_name: str) -> int:
+                config (str): The configuration name to load.
+        load_config(config: int) -> dict:
+            Loads the specified configuration file.
+                config (str): The constant id of the default configuration file to load.
+            Returns:
+                dict: The loaded configuration as a dictionary.
+    """
+
+    LEGACY_PMC = 1
+    PMC = 2
+    PLOS_GENETICS = 3
+    NATURE_GENETICS = 4
+
+    @staticmethod
+    def string_to_constant(config_name: str) -> int:
+        config_name = config_name.upper()
+        if hasattr(DefaultConfig, config_name):
+            return getattr(DefaultConfig, config_name)
+        raise ValueError(
+            f"Invalid default config name: {config_name}. Please check documentation for a list of default configs"
+        )
+
+    @staticmethod
+    def load_config(config: int) -> dict:
+        if config == DefaultConfig.LEGACY_PMC:
+            config_file = "config_pmc_pre_oct_2024.json"
+        elif config == DefaultConfig.PMC:
+            config_file = "config_pmc.json"
+        elif config == DefaultConfig.PLOS_GENETICS:
+            config_file = "config_plos_genetics.json"
+        elif config == DefaultConfig.NATURE_GENETICS:
+            config_file = "config_nature_genetics.json"
+        else:
+            raise Exception(
+                "A valid config was not provided. Please provide a valid DefaultConfig setting."
+            )
+
+        config_path = Path(__file__).parent / config_file
path: autocorpus/configs/default_config.py
user: login: Thomas-Rowlands
body: good point, I'm not familiar with the module before your code currently used so its not fresh on the mind
created_at: 2025-03-19T19:57:28Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,56 @@
+import json
+from pathlib import Path
+
+
+class DefaultConfig:
+    """
+    DefaultConfig class provides a way to initialize and load different configurations for the application.
+    Attributes:
+        LEGACY_PMC (int): Identifier for the legacy PMC configuration.
+        PMC (int): Identifier for the PMC configuration.
+        PLOS_GENETICS (int): Identifier for the PLOS Genetics configuration.
+        NATURE_GENETICS (int): Identifier for the Nature Genetics configuration.
+    Methods:
+        string_to_constant(config_name: str) -> int:
+                config (str): The configuration name to load.
+        load_config(config: int) -> dict:
+            Loads the specified configuration file.
+                config (str): The constant id of the default configuration file to load.
+            Returns:
+                dict: The loaded configuration as a dictionary.
+    """
+
+    LEGACY_PMC = 1
+    PMC = 2
+    PLOS_GENETICS = 3
+    NATURE_GENETICS = 4
+
+    @staticmethod
+    def string_to_constant(config_name: str) -> int:
+        config_name = config_name.upper()
+        if hasattr(DefaultConfig, config_name):
+            return getattr(DefaultConfig, config_name)
+        raise ValueError(
+            f"Invalid default config name: {config_name}. Please check documentation for a list of default configs"
+        )
path: autocorpus/configs/default_config.py
user: login: Thomas-Rowlands
body: Didn't think of that! But yes 
created_at: 2025-03-19T20:05:14Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,56 @@
+import json
+from pathlib import Path
+
+
+class DefaultConfig:
+    """
+    DefaultConfig class provides a way to initialize and load different configurations for the application.
+    Attributes:
+        LEGACY_PMC (int): Identifier for the legacy PMC configuration.
+        PMC (int): Identifier for the PMC configuration.
+        PLOS_GENETICS (int): Identifier for the PLOS Genetics configuration.
+        NATURE_GENETICS (int): Identifier for the Nature Genetics configuration.
+    Methods:
+        string_to_constant(config_name: str) -> int:
+                config (str): The configuration name to load.
+        load_config(config: int) -> dict:
+            Loads the specified configuration file.
+                config (str): The constant id of the default configuration file to load.
+            Returns:
+                dict: The loaded configuration as a dictionary.
+    """
+
+    LEGACY_PMC = 1
+    PMC = 2
+    PLOS_GENETICS = 3
+    NATURE_GENETICS = 4
path: autocorpus/configs/default_config.py
user: login: alexdewar
body: Ah ok -- for a library I agree that an enum makes more sense, so let's keep it.

I wasn't suggesting changing the constants to `LEGACY_PMC = "config_pmc.json"`, but I think that's a good idea! Then you can just get the file names as the values of the enum
created_at: 2025-03-20T10:47:02Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,56 @@
+import json
+from pathlib import Path
+
+
+class DefaultConfig:
+    """
+    DefaultConfig class provides a way to initialize and load different configurations for the application.
+    Attributes:
+        LEGACY_PMC (int): Identifier for the legacy PMC configuration.
+        PMC (int): Identifier for the PMC configuration.
+        PLOS_GENETICS (int): Identifier for the PLOS Genetics configuration.
+        NATURE_GENETICS (int): Identifier for the Nature Genetics configuration.
+    Methods:
+        string_to_constant(config_name: str) -> int:
+                config (str): The configuration name to load.
+        load_config(config: int) -> dict:
+            Loads the specified configuration file.
+                config (str): The constant id of the default configuration file to load.
+            Returns:
+                dict: The loaded configuration as a dictionary.
+    """
+
+    LEGACY_PMC = 1
+    PMC = 2
+    PLOS_GENETICS = 3
+    NATURE_GENETICS = 4
+
+    @staticmethod
+    def string_to_constant(config_name: str) -> int:
+        config_name = config_name.upper()
+        if hasattr(DefaultConfig, config_name):
+            return getattr(DefaultConfig, config_name)
+        raise ValueError(
+            f"Invalid default config name: {config_name}. Please check documentation for a list of default configs"
+        )
path: autocorpus/configs/default_config.py
user: login: Thomas-Rowlands
body: ok so I made another refactor after going through your linked Enums docs, did some digging on the laziest possible way of users retrieving config data directly.
created_at: 2025-03-24T13:20:59Z
author_association: COLLABORATOR
diff_hunk: @@ -160,7 +164,7 @@ def main():
     args = parser.parse_args()
     file_path = Path(args.filepath)
     target_dir = Path(args.target_dir if args.target_dir else "autoCORPus_output")
-    config = args.config
+    config = args.config if args.config else args.default_config
path: autocorpus/__main__.py
user: login: Thomas-Rowlands
body: this has since changed with my latest refactor
created_at: 2025-03-24T13:55:16Z
author_association: COLLABORATOR
diff_hunk: @@ -371,44 +381,23 @@ def __init__(
         """Utilises the input config file to create valid BioC versions of input HTML journal articles.
 
         Args:
-            config_path (str): path to the config file to be used
             base_dir (str): base directory of the input HTML journal articles
             main_text (str): path to the main text of the article (HTML files only)
             linked_tables (list): list of linked table file paths to be included in this run (HTML files only)
             table_images (list): list of table image file paths to be included in this run (JPEG or PNG files only)
             associated_data_path (str): currently unused
-            trained_data (str): currently unused, previously added for image processing
         """
-        # handle common
-        config = self.__read_config(config_path)
         self.base_dir = base_dir
         self.file_path = main_text
+        self.linked_tables = linked_tables
+        self.table_images = table_images
+        self.associated_data_path = associated_data_path
         self.main_text = {}
         self.empty_tables = {}
         self.tables = {}
         self.abbreviations = {}
         self.has_tables = False
-
-        # handle main_text
-        if self.file_path:
-            soup = self.__handle_html(self.file_path, config)
-            self.main_text = self.__extract_text(soup, config)
-            try:
-                self.abbreviations = Abbreviations(
-                    self.main_text, soup, config, self.file_path
-                ).to_dict()
-            except Exception as e:
-                print(e)
-        if linked_tables:
-            for table_file in linked_tables:
-                soup = self.__handle_html(table_file, config)
-        if table_images:
-            self.tables = TableImage(
-                table_images, self.base_dir, trained_data=trained_data
-            ).to_dict()
path: autocorpus/Autocorpus.py
user: login: Thomas-Rowlands
body: image OCR is currently not implemented and likely to change based on how the package looks after all of these refactors, so I left it out to make it less cluttered.
created_at: 2025-03-24T13:56:35Z
author_association: COLLABORATOR
diff_hunk: @@ -203,15 +207,27 @@ def main():
             )
             base_dir = file_path.parent if not file_path.is_dir() else file_path
             try:
+                if args.config:
+                    config = Autocorpus.read_config(args.config)
+                elif args.default_config:
+                    try:
+                        config = DefaultConfig[args.default_config]
path: autocorpus/__main__.py
user: login: AdrianDAlessandro
body: This means the `config` object that is input to the `Autocorpus` constructor could be either `dict` type or a `DefaultConfig` type. It'd be better to just keep it as a dictionary object. This will also have the benefit of not needing to set all of the extra methods (`__repr__`, `__get_item__`, `values`, etc) on the Enum definition.

Do something like:

```python
config = DefaultConfig[args.default_config].load_config()
```
created_at: 2025-03-24T17:58:01Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,55 @@
+import importlib.resources as resources
+import json
+from enum import Enum
+
+
+class DefaultConfig(Enum):
+    """DefaultConfig(Enum):
+        An enumeration for default configuration files used in the Auto-CORPus project.
+
+    Attributes:
+            LEGACY_PMC (dict): Legacy PMC configuration (pre-October 2024).
+            PMC (dict): Current PMC configuration.
+            PLOS_GENETICS (dict): PLOS Genetics configuration.
+            NATURE_GENETICS (dict): Nature Genetics configuration.
path: autocorpus/configs/default_config.py
user: login: alexdewar
body: These aren't dicts anymore
created_at: 2025-03-24T19:04:13Z
author_association: COLLABORATOR
diff_hunk: @@ -203,15 +207,27 @@ def main():
             )
             base_dir = file_path.parent if not file_path.is_dir() else file_path
             try:
+                if args.config:
+                    config = Autocorpus.read_config(args.config)
+                elif args.default_config:
+                    try:
+                        config = DefaultConfig[args.default_config]
path: autocorpus/__main__.py
user: login: Thomas-Rowlands
body: Just pushed the update for this, cheers
created_at: 2025-03-25T10:25:06Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,55 @@
+import importlib.resources as resources
+import json
+from enum import Enum
+
+
+class DefaultConfig(Enum):
+    """DefaultConfig(Enum):
+        An enumeration for default configuration files used in the Auto-CORPus project.
+
+    Attributes:
+            LEGACY_PMC (dict): Legacy PMC configuration (pre-October 2024).
+            PMC (dict): Current PMC configuration.
+            PLOS_GENETICS (dict): PLOS Genetics configuration.
+            NATURE_GENETICS (dict): Nature Genetics configuration.
path: autocorpus/configs/default_config.py
user: login: Thomas-Rowlands
body: Corrected with this new update
created_at: 2025-03-25T10:25:22Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,43 @@
+import importlib.resources as resources
+import json
+from enum import Enum
+
+
+class DefaultConfig(Enum):
+    """DefaultConfig(Enum):
+        An enumeration representing different configuration files for various datasets.
path: autocorpus/configs/default_config.py
user: login: AdrianDAlessandro
body: This should fix the ruff error. Along with adding a docstring to the top of the file.

```suggestion
    """An enumeration representing different configuration files for various datasets.
```
created_at: 2025-03-25T11:14:59Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,43 @@
+import importlib.resources as resources
+import json
+from enum import Enum
+
+
+class DefaultConfig(Enum):
+    """DefaultConfig(Enum):
+        An enumeration representing different configuration files for various datasets.
path: autocorpus/configs/default_config.py
user: login: Thomas-Rowlands
body: Thanks, I was getting triggered
created_at: 2025-03-25T11:15:44Z
author_association: COLLABORATOR

PR comments:
user: login: Thomas-Rowlands
created_at: 2025-03-24T13:59:56Z
body: Re-requested reviews as plenty of requested changes have been implemented but one may have been missed, I've put a refactor of the DefaultConfig class in which outdated some of the issues
author_association: COLLABORATOR
user: login: Thomas-Rowlands
created_at: 2025-03-25T10:26:07Z
body: load_config is now called to retrieve the config data as a dict, with the class docs adjusted
author_association: COLLABORATOR
user: login: alexdewar
created_at: 2025-03-25T10:43:42Z
body: Oh, except that `ruff` is complaining about missing docstrings. You'll need to fix that before it'll let you merge.

Tip: you can install a hook locally to check this stuff before you commit if you run `pre-commit install`.
author_association: COLLABORATOR
user: login: Thomas-Rowlands
created_at: 2025-03-25T10:55:49Z
body: > Oh, except that `ruff` is complaining about missing docstrings. You'll need to fix that before it'll let you merge.
> 
> Tip: you can install a hook locally to check this stuff before you commit if you run `pre-commit install`.

Just tried my local ruff and its not reporting what is listed in the pre-commit.ci check on GitHub, so looks like I have a config issue. I've made changes based on the listed problems in the ruff GitHub action so far
author_association: COLLABORATOR