Repository: intel/dffml

Issue:
source: ini: Create source for parsing .ini files
The goal of this issue is to show how to create a source derived from `FileSource` and `MemorySource` which saves and loads data from `.ini` files.

To do this we will create a new source, `INISource` (the call to `entrypoint` will be `@entrypoint("ini")`).

The source will be similar in structure to the `IDX1` and `IDX3` sources. Only we'll be implementing the `dump_fd` method in this one.

The [`configparser`](https://docs.python.org/3/library/configparser.html) library will be helpful here, since it parses `.ini` files.

Since we're working with file objects in `load_fd` and `dump_fd`, rather than filenames, we'll want to use the [`read_file`](https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.read_file) method within `load_fd` .

In `dump_fd`, we'll likely want to use [`read_dict`](https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.read_dict) to create a `Configparser`. Both `load_fd` and `dump_fd` should have local variables which are an instance of `Configparser`. We don't want to make it class local because if features in an individual record are deleted then they would still show up in the output.

You'll want to make sure to call `record.features()` and using that as the data to save for the record. Also assume when loading that you're loading the feature data. Each `[section]` should be used as a record. With the name of the section being the `key` for that record.

Make sure to write lots of comments, since this will be used as the example for another tutorial on Sources (see issue https://github.com/intel/dffml/issues/282).

Here is the `idx1` source as an example.

https://github.com/intel/dffml/blob/155c114da35b80d5a17c96eece5c45dc8a6322c5/dffml/source/idx1.py#L6-L56


Discussion:

Resolving patch:
diff --git a/dffml/source/ini.py b/dffml/source/ini.py
new file mode 100644
--- /dev/null
+++ b/dffml/source/ini.py
@@ -0,0 +1,66 @@
+from configparser import ConfigParser
+
+from ..base import config
+from ..record import Record
+from .file import FileSource
+from .memory import MemorySource
+from ..util.data import parser_helper
+from ..util.entrypoint import entrypoint
+
+
+@config
+class INISourceConfig:
+    filename: str
+    readwrite: bool = False
+    allowempty: bool = False
+
+
+@entrypoint("ini")
+class INISource(FileSource, MemorySource):
+    """
+    Source to read files in .ini format.
+    """
+
+    CONFIG = INISourceConfig
+
+    async def load_fd(self, ifile):
+        # Creating an instance of configparser
+        parser = ConfigParser()
+        # Read from a file object
+        parser.read_file(ifile)
+        # Get all the sections present in the file
+        sections = parser.sections()
+
+        self.mem = {}
+
+        # Go over each section
+        for section in sections:
+            # Get data under each section as a dict
+            temp_dict = {}
+            for k, v in parser.items(section):
+                temp_dict[k] = parser_helper(v)
+            # Each section used as a record
+            self.mem[str(section)] = Record(
+                str(section), data={"features": temp_dict},
+            )
+
+        self.logger.debug("%r loaded %d sections", self, len(self.mem))
+
+    async def dump_fd(self, fd):
+        # Create an instance of configparser
+        parser = ConfigParser()
+
+        # Go over each section and record in mem
+        for section, record in self.mem.items():
+            # Get each section data as a dict
+            section_data = record.features()
+            if section not in parser.keys():
+                # If section does not exist add new section
+                parser.add_section(section)
+            # Set section data
+            parser[section] = section_data
+
+        # Write to the fileobject
+        parser.write(fd)
+
+        self.logger.debug("%r saved %d sections", self, len(self.mem))
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -77,6 +77,7 @@
             "idx1 = dffml.source.idx1:IDX1Source",
             "idx3 = dffml.source.idx3:IDX3Source",
             "db = dffml.source.db:DbSource",
+            "ini = dffml.source.ini:INISource",
         ],
         "dffml.port": ["json = dffml.port.json:JSON"],
         "dffml.service.cli": ["dev = dffml.service.dev:Develop"],


Test patch:
diff --git a/tests/source/test_ini.py b/tests/source/test_ini.py
new file mode 100644
--- /dev/null
+++ b/tests/source/test_ini.py
@@ -0,0 +1,30 @@
+import os
+from tempfile import TemporaryDirectory
+
+from dffml.record import Record
+from dffml.source.ini import INISource
+from dffml.high_level import load, save
+from dffml.util.asynctestcase import AsyncTestCase
+
+
+class TestINISource(AsyncTestCase):
+    async def test_ini(self):
+        with TemporaryDirectory() as testdir:
+            self.testfile = os.path.join(testdir, "testfile.ini")
+            # Create a source
+            source = INISource(
+                filename=self.testfile, allowempty=True, readwrite=True,
+            )
+            # Save some data in the source
+            await save(
+                source,
+                Record("section1", data={"features": {"A": 1, "B": 2,}}),
+                Record("section2", data={"features": {"C": 3, "D": 4,}}),
+            )
+            # Load all the records
+            records = [record async for record in load(source)]
+
+            self.assertIsInstance(records, list)
+            self.assertEqual(len(records), 2)
+            self.assertDictEqual(records[0].features(), {"a": 1, "b": 2})
+            self.assertDictEqual(records[1].features(), {"c": 3, "d": 4})