Repository: rseng/rse (language: python)

Issue:
Add a generic CsvImporter?
This is discussion / feature request, not a bug

Related to discussion here:
https://github.com/rhine3/bioacoustics-software/issues/3#issuecomment-1240151849

> I wonder if it would make sense to add an importer/scraper that can take a .csv directly? This way the data could be munged + cleaned, e.g. by a script, before handing it off to rse.

It seems like this might be possible by decoupling some of the logic in the google sheets scraper?

Like maybe make a mixin that does the specific sheets accessing part, and have a more generic csv parser class that could then be combined with another mixin that parses a csv from a local file

So you'd have something like

```python
class SpreadSheetImporter(ScraperBase):
    def parse(self, f, args):
        # csv parsing logic that's in GoogleSheetImporter now
        # starting with https://github.com/rseng/rse/blob/299ee56d33e4b9868deef4d624cca0608a786cbf/rse/main/scrapers/googlesheet.py#L58
        reader = csv.reader(f, delimiter=",")
        ...
        return self.results


class GoogleSheetMixIn:
    def scrape(self, args):
        ...
        f = StringIO(response.text)
        return f


class GoogleSheetImporter(SpreadSheetImporter, GoogleSheetMixin):
    ...
   def scrape(self, args):
       f = self.scrape(args)
       self.parse(f, args)


class CsvMixIn:
    def load(self, csv_path):
        ...
        f = pandas.read_csv(csv_path)


class CsvImporter(ScraperBase, CsvMixIn):
    ...
    def scrape(self, args):
        
```
I know this really starts to stretch the definition of "scrape" but I hope you see what I'm suggesting here in terms of decoupling the functionality



Discussion / hints:
yep that would be very easy to do! All of the above looks reasonable, but we'd use the native csv module (not pandas, that would add a dependency that isn't necessary) and I wouldn't call them Mixins, but just class names for what they do!

I'll add to my queue of TODOs, thanks for the idea!

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a1682f2..3be2933 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,6 +14,7 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip.
 
 ## [0.0.x](https://github.com/rseng/rse/tree/master) (0.0.x)
+ - support for csv import (0.0.46)
  - ensure Google scraper skips malformed rows, etc (0.0.45)
  - Logging bugs and adding export/import docs (0.0.44)
  - Adding debian-med scraper (0.0.43)
diff --git a/docs/_docs/getting-started/commands.md b/docs/_docs/getting-started/commands.md
index 6d7e707..d5c1a3a 100644
--- a/docs/_docs/getting-started/commands.md
+++ b/docs/_docs/getting-started/commands.md
@@ -354,19 +354,44 @@ As a set of examples, here are how titles might be translated into uids and path
 
 If you find a title that doesn't parse well or would like to request a new kind of import, please [open an issue](https://github.com/rseng/rse). Each importer is described in more detail below.
 
-<a id="import-google-spreadsheet">
-### Google Sheet
+<a id="import-csv">
+### CSV
 
-It might be the case that you want to have software input from a form, and then input via a Google sheet.
-We provide an [example template sheet](https://docs.google.com/spreadsheets/d/1ZW2kOsBOfSpRSH_9Efvz-ytn7dQ2m1DmYDBdIVNGY4c/edit?usp=sharing) that you could provide to `rse import` with the `google-sheet` and path to it's exported csv. For your
-spreadsheet, the following fields are required:
+A csv import takes a basic csv (or other deliminated) file and imports it. If you don't want to use the Google Sheets
+importer, or want to inspect your csv first, this could be an option. The following fields (first row) are required:
 
  - Title: (required) A human-friendly title to describe the software. If the Url doesn't have a version control address, this will be parsed for a unique identifier.
  - Url: (required) A link to GitHub, GitLab, or another online resource (this will be parsed looking for a unique identifier)
  - Description: (required) The description of the software project
  - Tags: (optional) A list of tags, comma separated, to parse into the metadata.
 
-All other fields will be treated as custom data fields to add to the data. Once you have your data sheet, you'll want to make
+Here is an example import
+
+```bash
+$ mkdir bioacousics
+$ cd bioacousics
+$ rse init .
+INFO:rse.main.config:Generating configuration file rse.ini
+```
+We can then run the import:
+
+```bash
+$ rse import --type csv software-sheet.csv
+```
+
+You can also tweak the newline parameter, and delimiter (default `,`).
+
+```bash
+$ rse import --type csv --delim="," software-sheet.csv
+```
+
+
+<a id="import-google-spreadsheet">
+### Google Sheet
+
+It might be the case that you want to have software input from a form, and then input via a Google sheet.
+We provide an [example template sheet](https://docs.google.com/spreadsheets/d/1ZW2kOsBOfSpRSH_9Efvz-ytn7dQ2m1DmYDBdIVNGY4c/edit?usp=sharing) that you could provide to `rse import` with the `google-sheet` and path to it's exported csv. The fields (first column) required
+are the same as for the CSV import detailed above. Once you have your data sheet, you'll want to make
 sure to generate a public link to export csv. You can do that via:
 
 ```
diff --git a/rse/client/__init__.py b/rse/client/__init__.py
index 146e179..e4af8bf 100644
--- a/rse/client/__init__.py
+++ b/rse/client/__init__.py
@@ -169,7 +169,7 @@ def get_parser():
         dest="import_type",
         help="Type to import (defaults to google-sheet)",
         default="google-sheet",
-        choices=["google-sheet"],
+        choices=["google-sheet", "csv"],
     )
 
     imp.add_argument(
diff --git a/rse/client/imp.py b/rse/client/imp.py
index 18ea0d5..097a266 100644
--- a/rse/client/imp.py
+++ b/rse/client/imp.py
@@ -13,6 +13,25 @@
 from rse.main.scrapers import get_importer
 
 
+def parse_extra(extra):
+    """
+    Allow for basic parsing of extra args, in form <arg>=<value> only.
+    """
+    args = []
+    kwargs = {}
+    for arg in extra:
+
+        # This is a key value pair (extra)
+        if arg.startswith("--") and "=" in arg:
+            key, val = arg.strip().split("=")
+            kwargs[key.strip()] = val.strip()
+
+        # This is an entity to import
+        elif not arg.startswith("--"):
+            args.append(arg)
+    return args, kwargs
+
+
 def main(args, extra):
 
     try:
@@ -20,6 +39,9 @@ def main(args, extra):
     except:
         sys.exit(f"{args.import_type} is not a known importer.")
 
+    # Ensure support for --delim and --newline
+    extra, kwargs = parse_extra(extra)
+
     results = importer.scrape(extra)
     print("Found %s results" % len(results))
 
diff --git a/rse/main/scrapers/__init__.py b/rse/main/scrapers/__init__.py
index a620281..ac142ed 100644
--- a/rse/main/scrapers/__init__.py
+++ b/rse/main/scrapers/__init__.py
@@ -14,6 +14,7 @@
 from .ascl import AsclScraper
 from .biogrids import BioGridsScraper
 from .biotools import BioToolsScraper
+from .csv import CSVImporter
 from .debian import DebianMedScraper
 from .googlesheet import GoogleSheetImporter
 from .hal import HalScraper
@@ -43,6 +44,8 @@ def get_importer(name):
     An importer is technically a kind of scraper.
     """
     importer = None
+    if name.lower() == "csv":
+        importer = CSVImporter()
     if name.lower() == "google-sheet":
         importer = GoogleSheetImporter()
     if not importer:
diff --git a/rse/main/scrapers/csv.py b/rse/main/scrapers/csv.py
new file mode 100644
index 0000000..ba1deba
--- /dev/null
+++ b/rse/main/scrapers/csv.py
@@ -0,0 +1,136 @@
+"""
+
+Copyright (C) 2022 Vanessa Sochat.
+
+This Source Code Form is subject to the terms of the
+Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
+with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+"""
+
+import csv
+import logging
+import os
+import sys
+
+from rse.main.parsers import CustomParser, get_parser
+from rse.utils.strings import update_nonempty
+
+from .base import ScraperBase
+
+bot = logging.getLogger("rse.main.import.csv")
+
+
+def read_rows(filepath, newline=None, delim=None):
+    """
+    read in the data rows of a csv file.
+    """
+    newline = newline or ""
+    delim = delim or ","
+
+    # Read in the entire membership counts
+    with open(filepath, newline=newline) as infile:
+        reader = csv.reader(infile, delimiter=delim)
+        data = [row for row in reader]
+    return data
+
+
+class CSVImporter(ScraperBase):
+
+    name = "csv"
+
+    def scrape(self, args, **kwargs):
+        """
+        The main function to perform the local "scrape" to return the csv
+        """
+        delim = kwargs.get("delim")
+        newline = kwargs.get("newline")
+
+        # should add ccustom args here
+        # We expect a listing of paths
+        if isinstance(args, str):
+            args = [args]
+
+        # Each arg should be a URL
+        for sheet in args:
+            if not os.path.exists(sheet):
+                bot.info(f"{sheet} does not exist.")
+                continue
+            bot.info(f"Reading sheet {sheet}")
+            rows = read_rows(sheet, delim=delim, newline=newline)
+            if not rows:
+                bot.warning(f"Sheet {sheet} does not have any rows.")
+                continue
+            self.results += self.parse_rows(rows)
+        return self.results
+
+    def parse_rows(self, rows):
+        """
+        Shared function to parse rows.
+        """
+        data = []
+        if not rows:
+            return data
+
+        # Prepare based on header
+        header = [x.lower() for x in rows.pop(0)]
+        for required in ["title", "url", "description"]:
+            if required not in header:
+                sys.exit(f"Sheet is missing required field {required}.")
+
+        for row in rows:
+            repo = {value: row[i] for i, value in enumerate(header) if value}
+            complete = True
+            for required in ["title", "url", "description"]:
+                if not repo[required]:
+                    bot.warning("Row {row} is not complete, missing field {required}.")
+                    complete = False
+            if not complete:
+                continue
+
+            # If we have topics, ensure to parse
+            if "tags" in repo:
+                repo["tags"] = [x.strip() for x in repo["tags"].split(",")]
+            bot.info("Found software record: %s" % repo["url"])
+            data.append(repo)
+        return data
+
+    def create(self, database=None, config_file=None, update=False):
+        """
+        After a scrape (whether we obtain latest or a search query) we
+        run create to create software repositories based on results.
+        """
+        from rse.main import Encyclopedia
+
+        client = Encyclopedia(config_file=config_file, database=database)
+        for result in self.results:
+            uid = result["url"].split("//")[-1]
+
+            # If a repository is added that isn't represented
+            try:
+                repo = get_parser(uid)
+                data = repo.get_metadata()
+
+                # Empty or malformed repository
+                if not data:
+                    bot.warning(f"Skipping malformed entry {uid}")
+                    continue
+                result = update_nonempty(result, data)
+
+            # Or as custom entry
+            except NotImplementedError:
+                # Base UID based on title
+                repo = CustomParser(result["title"])
+                repo.set_metadata(
+                    title=result["title"],
+                    url=result["url"],
+                    description=result["description"],
+                )
+
+            # Add results that don't exist
+            exists = client.exists(repo.uid)
+            if not exists:
+                client.add(repo.uid, data=result)
+            elif exists and update:
+                # rewrite defaults to false
+                client.update(repo.uid, data=result)
diff --git a/rse/main/scrapers/googlesheet.py b/rse/main/scrapers/googlesheet.py
index 89dd423..883cfc8 100644
--- a/rse/main/scrapers/googlesheet.py
+++ b/rse/main/scrapers/googlesheet.py
@@ -15,31 +15,18 @@
 
 import requests
 
-from rse.main.parsers import CustomParser, get_parser
-from rse.utils.strings import update_nonempty
 from rse.utils.urls import get_user_agent
 
-from .base import ScraperBase
+from .csv import CSVImporter
 
 bot = logging.getLogger("rse.main.import.google-sheet")
 
 
-def read_rows(filepath, newline="", delim=","):
-    """
-    read in the data rows of a csv file.
-    """
-    # Read in the entire membership counts
-    with open(filepath, newline=newline) as infile:
-        reader = csv.reader(infile, delimiter=delim)
-        data = [row for row in reader]
-    return data
-
-
-class GoogleSheetImporter(ScraperBase):
+class GoogleSheetImporter(CSVImporter):
 
     name = "googlesheet"
 
-    def scrape(self, args):
+    def scrape(self, args, **kwargs):
         """
         The main function to perform the scrape to import and return results.
         """
@@ -59,64 +46,5 @@ def scrape(self, args):
             rows = [x for x in reader if x]
             if not rows:
                 sys.exit(f"Sheet {url} does not have any rows.")
-
-            header = [x.lower() for x in rows.pop(0)]
-            for required in ["title", "url", "description"]:
-                if required not in header:
-                    sys.exit(f"Sheet {url} is missing required field {required}.")
-
-            for row in rows:
-                repo = {value: row[i] for i, value in enumerate(header) if value}
-                complete = True
-                for required in ["title", "url", "description"]:
-                    if not repo[required]:
-                        complete = False
-                if not complete:
-                    continue
-
-                # If we have topics, ensure to parse
-                if "tags" in repo:
-                    repo["tags"] = [x.strip() for x in repo["tags"].split(",")]
-                bot.info("Found software record: %s" % repo["url"])
-                self.results.append(repo)
-        return self.results
-
-    def create(self, database=None, config_file=None, update=False):
-        """
-        After a scrape (whether we obtain latest or a search query) we
-        run create to create software repositories based on results.
-        """
-        from rse.main import Encyclopedia
-
-        client = Encyclopedia(config_file=config_file, database=database)
-        for result in self.results:
-            uid = result["url"].split("//")[-1]
-
-            # If a repository is added that isn't represented
-            try:
-                repo = get_parser(uid)
-                data = repo.get_metadata()
-
-                # Empty or malformed repository
-                if not data:
-                    bot.warning(f"Skipping malformed entry {uid}")
-                    continue
-                result = update_nonempty(result, data)
-
-            # Or as custom entry
-            except NotImplementedError:
-                # Base UID based on title
-                repo = CustomParser(result["title"])
-                repo.set_metadata(
-                    title=result["title"],
-                    url=result["url"],
-                    description=result["description"],
-                )
-
-            # Add results that don't exist
-            exists = client.exists(repo.uid)
-            if not exists:
-                client.add(repo.uid, data=result)
-            elif exists and update:
-                # rewrite defaults to false
-                client.update(repo.uid, data=result)
+            self.results += self.parse_rows(rows)
+            return self.results
diff --git a/rse/version.py b/rse/version.py
index 4a344ca..3672cd6 100644
--- a/rse/version.py
+++ b/rse/version.py
@@ -9,7 +9,7 @@
 """
 
 
-__version__ = "0.0.45"
+__version__ = "0.0.46"
 AUTHOR = "Vanessa Sochat"
 AUTHOR_EMAIL = "vsoch@users.noreply.github.io"
 NAME = "rse"


Test patch:
diff --git a/tests/data/software-sheet.csv b/tests/data/software-sheet.csv
new file mode 100644
index 0000000..b953426
--- /dev/null
+++ b/tests/data/software-sheet.csv
@@ -0,0 +1,10 @@
+Title,Url,Description,Tags,"Paid, free, open",Code available,License,"Taxonomic or sound type specialization, if any",Type,Data exploration,Organization and annotation,Sound processing features,Acoustic parameter measurement,Localization,Detector features,Classifier features,Metadata standard,Comments
+Acoustic Indices,https://github.com/patriceguyot/Acoustic_Indices,calculate acoustic indices,acousic,free,Y,GPL v3,,Python,,,,extract acoustic indices to use as biodiversity proxy,,,,,
+Adobe audition,https://www.adobe.com/products/audition.html,geared toward sound editing,editing,paid,N,,,desktop GUI,general-purpose audio review,,edit audio,,,,,,
+Anabat insight,https://www.titley-scientific.com/us/anabat-insight.html,"recording organization, review, and classification","record, review, classify",free and paid versions,N,,,desktop GUI,browse and view files as spectrograms,view labels on built-in maps,"filter and process files with time expansion, pitch shift, etc.","calculate pulse and pass acoustic parameters, e.g. max frequency",,bat call detection and custom filtering in full-spectrum and zero-crossing recordings,create decision tree classifiers and support third-party ID algorithms,GUANO,
+Animal Sound Identifier,https://datadryad.org/stash/dataset/doi:10.5061/dryad.221mq23,"classify animals using training data directly from field recordings, without reference libraries",classify,free,Y,cc0,,MATLAB,,,,,,,create cross-correlation classifiers; semi-active learning approach to create classifiers with little initial training data,,Related paper - https://onlinelibrary.wiley.com/doi/full/10.1111/ele.13092
+ANIMAL-SPOT,https://github.com/ChristianBergler/ANIMAL-SPOT,Python framework for building deep learning classification models,"classify,python",free,Y,GPL v3,,Python,,,,,,,create ResNet18 CNN classifiers,,
+Rainforest Connection Arbimon,https://arbimon.rfcx.org/,web-based interface geared toward analyzing autonomous wildlife survey data,web,free for certain number of recordings,N,,,"web GUI, phone app",listening and review interface for autonomous recordings; create listening playlists,web-based annotation GUI; phone app for managing recorder deployment; web uploader that supports many common recorder timestamp formats,,aggregate recordings at different temporal and spatial scales,,template-matching detection,,,free storage
+ARTWARP,https://soundanalysis.wp.st-andrews.ac.uk/,estimate frequency sweeps and cluster-classify tonal sounds,analysis,free,N,NO LICENCE,tonal sounds; individual animal recognition,MATLAB,,,,identify frequncy sweeps,,,create classifier of tonal sounds using ART neural network,,
+Audacity,https://www.audacityteam.org/download/,"simple, lightweight listening and spectrogram viewing, comparison, and manipulation",audio,free,Y,GPL v3,,desktop GUI,simple but powerful software for viewing one or many recording waveforms or spectrograms at once,,full suite of sound editing features,,,,,,
+github.com/nwolek/audiomoth-scripts,https://github.com/nwolek/audiomoth-scripts,spectrogram visualization and AudioMoth-specific tools,audio,free,Y,MIT,recordings collected from AudioMoth recorders,command line,a variety of spectrogram visualization bash scripts,script to rename AudioMoth hex files to human-readable filenames,,,,,,,
\ No newline at end of file
diff --git a/tests/test_database.py b/tests/test_database.py
index e4963c3..3f64cd0 100644
--- a/tests/test_database.py
+++ b/tests/test_database.py
@@ -9,6 +9,7 @@
 """
 
 import os
+
 import pytest
 
 
diff --git a/tests/test_import.py b/tests/test_import.py
index bd22abe..86685da 100644
--- a/tests/test_import.py
+++ b/tests/test_import.py
@@ -8,16 +8,37 @@
 
 """
 
+import os
+
 import pytest
 
 from rse.main.scrapers import get_importer
 
+here = os.path.dirname(os.path.abspath(__file__))
+
+data = os.path.join(here, "data")
+
 
 def test_get_unknown_import():
     with pytest.raises(SystemExit):
         get_importer("noodles")
 
 
+def test_csv_import(tmp_path):
+    """
+    Test csv import
+    """
+    sheet = os.path.join(data, "software-sheet.csv")
+    importer = get_importer("csv")
+    results = importer.scrape(sheet)
+    assert len(results) == 9
+
+    # Assert we have required fields
+    for result in results:
+        for key in ["title", "url", "description"]:
+            assert key in result and result[key]
+
+
 def test_google_sheet_import(tmp_path):
     """
     Test google-sheet import