Repository: urlstechie/urlchecker-python

Issue:
additional feature: save to file
The library should implement an option to export results to file, which will support https://github.com/urlstechie/URLs-checker/issues/23

Comments:

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index dafc657..077dfa1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,7 @@ and **Merged pull requests**. Critical items to know are:
 Referenced versions in headers are tagged on Github, in parentheses are for pypi.
 
 ## [vxx](https://github.com/urlstechie/urlschecker-python/tree/master) (master)
+ - adding support for csv export (0.0.12)
  - fixing bug with parameter type for retry count and timeout (0.0.11)
  - first release of urlchecker module with container, tests, and brief documentation (0.0.1)
  - dummy release for pypi (0.0.0)
diff --git a/README.md b/README.md
index 5dc6652..c419fbc 100644
--- a/README.md
+++ b/README.md
@@ -156,6 +156,8 @@ https://stackoverflow.com/questions/49197916/how-to-profile-cpu-usage-of-a-pytho
 Done. All URLS passed.
 ```
 
+### Check GitHub Repository
+
 But wouldn't it be easier to not have to clone the repository first?
 Of course! We can specify a GitHub url instead, and add `--cleanup`
 if we want to clean up the folder after.
@@ -171,6 +173,100 @@ sure that you provide a comma separated list *without any spaces*
 urlchecker check --white-listed-files=README.md,_config.yml
 ```
 
+### Save Results
+
+If you want to save your results to file, perhaps for some kind of record or
+other data analysis, you can provide the `--save` argument:
+
+```bash
+$ urlchecker check --save results.csv .
+  original path: .
+     final path: /home/vanessa/Desktop/Code/urlstechie/urlchecker-test-repo
+      subfolder: None
+         branch: master
+        cleanup: False
+     file types: ['.md', '.py']
+      print all: True
+ url whitetlist: []
+   url patterns: []
+  file patterns: []
+     force pass: False
+    retry count: 2
+           save: results.csv
+        timeout: 5
+
+ /home/vanessa/Desktop/Code/urlstechie/urlchecker-test-repo/README.md 
+ --------------------------------------------------------------------
+No urls found.
+
+ /home/vanessa/Desktop/Code/urlstechie/urlchecker-test-repo/test_files/sample_test_file.py 
+ -----------------------------------------------------------------------------------------
+https://github.com/SuperKogito/URLs-checker/README.md
+https://github.com/SuperKogito/URLs-checker/README.md
+https://www.google.com/
+https://github.com/SuperKogito
+
+ /home/vanessa/Desktop/Code/urlstechie/urlchecker-test-repo/test_files/sample_test_file.md 
+ -----------------------------------------------------------------------------------------
+https://github.com/SuperKogito/URLs-checker/blob/master/README.md
+https://github.com/SuperKogito/Voice-based-gender-recognition/issues
+https://github.com/SuperKogito/spafe/issues/7
+https://github.com/SuperKogito/URLs-checker
+https://github.com/SuperKogito/URLs-checker/issues
+https://github.com/SuperKogito/spafe/issues/4
+https://github.com/SuperKogito/URLs-checker/issues/2
+https://github.com/SuperKogito/URLs-checker/issues/2
+https://github.com/SuperKogito/Voice-based-gender-recognition/issues/1
+https://github.com/SuperKogito/spafe/issues/6
+https://github.com/SuperKogito/spafe/issues
+...
+
+Saving results to /home/vanessa/Desktop/Code/urlstechie/urlchecker-test-repo/results.csv
+
+
+Done. All URLS passed.
+```
+
+The file that you save to will include a comma separated value tabular listing
+of the urls, and their result. The result options are "passed" and "failed"
+and the default header is `URL,RESULT`. All of these defaults are exposed
+if you want to change them (e.g., using a tab separator or a different header)
+if you call the function from within Python. Here is an example of the default file
+produced, which should satisfy most use cases:
+
+```
+URL,RESULT
+https://github.com/SuperKogito,passed
+https://www.google.com/,passed
+https://github.com/SuperKogito/Voice-based-gender-recognition/issues,passed
+https://github.com/SuperKogito/Voice-based-gender-recognition,passed
+https://github.com/SuperKogito/spafe/issues/4,passed
+https://github.com/SuperKogito/Voice-based-gender-recognition/issues/2,passed
+https://github.com/SuperKogito/spafe/issues/5,passed
+https://github.com/SuperKogito/URLs-checker/blob/master/README.md,passed
+https://img.shields.io/,passed
+https://github.com/SuperKogito/spafe/,passed
+https://github.com/SuperKogito/spafe/issues/3,passed
+https://www.google.com/,passed
+https://github.com/SuperKogito,passed
+https://github.com/SuperKogito/spafe/issues/8,passed
+https://github.com/SuperKogito/spafe/issues/7,passed
+https://github.com/SuperKogito/Voice-based-gender-recognition/issues/1,passed
+https://github.com/SuperKogito/spafe/issues,passed
+https://github.com/SuperKogito/URLs-checker/issues,passed
+https://github.com/SuperKogito/spafe/issues/2,passed
+https://github.com/SuperKogito/URLs-checker,passed
+https://github.com/SuperKogito/spafe/issues/6,passed
+https://github.com/SuperKogito/spafe/issues/1,passed
+https://github.com/SuperKogito/URLs-checker/README.md,failed
+https://github.com/SuperKogito/URLs-checker/issues/3,failed
+https://none.html,failed
+https://github.com/SuperKogito/URLs-checker/issues/2,failed
+https://github.com/SuperKogito/URLs-checker/README.md,failed
+https://github.com/SuperKogito/URLs-checker/issues/1,failed
+https://github.com/SuperKogito/URLs-checker/issues/4,failed
+```
+
 If you have any questions, please don't hesitate to [open an issue](https://github.com/urlstechie/urlchecker-python).
 
 
diff --git a/urlchecker/client/__init__.py b/urlchecker/client/__init__.py
index 581448d..3fe8333 100755
--- a/urlchecker/client/__init__.py
+++ b/urlchecker/client/__init__.py
@@ -119,6 +119,13 @@ def get_parser():
         default="",
     )
 
+# Saving
+
+    check.add_argument(
+        "--save",
+        help="Path toa csv file to save results to.",
+        default=None,
+    )
 
 # Timeouts
 
diff --git a/urlchecker/client/check.py b/urlchecker/client/check.py
index 2786117..9a4ae51 100644
--- a/urlchecker/client/check.py
+++ b/urlchecker/client/check.py
@@ -9,7 +9,7 @@ import sys
 import logging
 
 from urlchecker.main.github import clone_repo, delete_repo
-from urlchecker.core.fileproc import remove_empty
+from urlchecker.core.fileproc import remove_empty, save_results
 from urlchecker.core.check import run_urlchecker
 from urlchecker.logger import print_success, print_failure
 
@@ -65,9 +65,9 @@ def main(args, extra):
     print("  file patterns: %s" % white_listed_files)
     print("     force pass: %s" % args.force_pass)
     print("    retry count: %s" % args.retry_count)
+    print("           save: %s" % args.save) 
     print("        timeout: %s" % args.timeout)
 
-
     # Run checks, get lookup of results and fails
     check_results = run_urlchecker(path=path,
                                    file_types=file_types,
@@ -78,6 +78,10 @@ def main(args, extra):
                                    retry_count=args.retry_count,
                                    timeout=args.timeout)
 
+    # save results to flie, if save indicated
+    if args.save:
+        save_results(check_results, args.save)
+
     # delete repo when done, if requested
     if args.cleanup:
         logger.info("Cleaning up %s..." % path)
diff --git a/urlchecker/core/fileproc.py b/urlchecker/core/fileproc.py
index 412bc76..4379f3d 100644
--- a/urlchecker/core/fileproc.py
+++ b/urlchecker/core/fileproc.py
@@ -7,8 +7,10 @@ For a copy, see <https://opensource.org/licenses/MIT>.
 
 """
 
-import re
+import csv
 import os
+import re
+import sys
 from urlchecker.core import urlmarker
 
 
@@ -119,3 +121,46 @@ def remove_empty(file_list):
         (list) list of (non None or empty string) contents.
     """
     return [x for x in file_list if x not in ["", None]]
+
+
+def save_results(check_results, file_path, sep=",", header=None):
+    """
+    Given a check_results dictionary, a dict with "failed" and "passed" keys (
+    or more generally, keys to indicate some status), save a csv
+    file that has header with URL,RESULT that indicates, for each url,
+    a pass or failure. If the directory of the file path doesn't exist, exit
+    on error.
+
+    Args:
+        - check_results (dict): the check results dictionary with passed/failed
+        - file_path (str): the file path (.csv) to save to.
+        - sep (str): the separate to use (defaults to comma)
+        - header (list): if not provided, will save URL,RESULT
+
+    Returns:
+        (str) file_path: a newly saved csv with the results
+    """
+    # Ensure that the directory exists
+    file_path = os.path.abspath(file_path)
+    dirname = os.path.dirname(file_path)
+
+    if not os.path.exists(dirname):
+        sys.exit("%s does not exist, cannot save %s there." %(dirname, file_path))
+
+    # Ensure the header is provided and correct (length 2)
+    if not header:
+        header = ["URL", "RESULT"]
+
+    if len(header) != 2:
+        sys.exit("Header must be length 2 to match size of data.")
+
+    print("Saving results to %s" % file_path)
+
+    # Write to file after header row
+    with open(file_path, mode='w') as fd:
+        writer = csv.writer(fd, delimiter=sep, quotechar='"', quoting=csv.QUOTE_MINIMAL)
+        writer.writerow(header)
+        for result, items in check_results.items():
+            [writer.writerow([item, result]) for item in items];
+
+    return file_path
diff --git a/urlchecker/version.py b/urlchecker/version.py
index b8df2a4..1dcf70c 100644
--- a/urlchecker/version.py
+++ b/urlchecker/version.py
@@ -1,14 +1,13 @@
 """
 
-Copyright (C) 2017-2020 Vanessa Sochat.
+Copyright (c) 2020 Ayoub Malek and Vanessa Sochat
 
-This Source Code Form is subject to the terms of the
-Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
-with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+This source code is licensed under the terms of the MIT license.  
+For a copy, see <https://opensource.org/licenses/MIT>.
 
 """
 
-__version__ = "0.0.11"
+__version__ = "0.0.12"
 AUTHOR = "Ayoub Malek, Vanessa Sochat"
 AUTHOR_EMAIL = "vsochat@stanford.edu"
 NAME = "urlchecker"


Test patch:
diff --git a/tests/test_check.py b/tests/test_check.py
index 9ac1e7f..00eca36 100644
--- a/tests/test_check.py
+++ b/tests/test_check.py
@@ -4,6 +4,7 @@ import os
 import sys
 import pytest
 import subprocess
+import tempfile
 import configparser
 from urlchecker.core.fileproc import get_file_paths
 from urlchecker.main.github import clone_repo, delete_repo, get_branch
@@ -109,6 +110,7 @@ def test_script(config_fname, cleanup, print_all, force_pass, rcount, timeout):
                           stdout=subprocess.PIPE,
                           stderr=subprocess.PIPE)
 
+    
 
 @pytest.mark.parametrize('local_folder_path', ['./tests/test_files'])
 @pytest.mark.parametrize('config_fname', ['./tests/_local_test_config.conf'])
@@ -182,3 +184,39 @@ def test_check_generally(retry_count):
         print("\n\nDone. All URLS passed.")
         if retry_count == 3:
             return True
+
+
+@pytest.mark.parametrize('save', [True])
+def test_save(save):
+
+    # init config parser
+    config = configparser.ConfigParser()
+    config.read('./tests/_local_test_config.conf')
+
+    # init env variables
+    path  = config['DEFAULT']["git_path_test_value"]
+    file_types = config['DEFAULT']["file_types_test_values"]
+    white_listed_urls = config['DEFAULT']["white_listed_test_urls"]
+    white_listed_patterns =  config['DEFAULT']["white_listed__test_patterns"]
+
+    # Generate command
+    cmd = ["urlchecker", "check", "--subfolder", "_project", "--file-types", file_types,
+           "--white-listed-files", "conf.py", "--white-listed-urls", white_listed_urls,
+           "--white-listed_patterns", white_listed_patterns]
+
+    # Write to file
+    if save:
+        output_csv = tempfile.NamedTemporaryFile(suffix=".csv", prefix="urlchecker-")
+        cmd += ["--save", output_csv.name]
+
+    # Add final path
+    cmd.append(path)
+
+    print(" ".join(cmd))
+    # excute script
+    pipe = subprocess.run(cmd,
+                          stdout=subprocess.PIPE,
+                          stderr=subprocess.PIPE)
+    if save:
+        if not os.path.exists(output_csv.name):
+            raise AssertionError
diff --git a/tests/test_fileproc.py b/tests/test_fileproc.py
index cba17f3..c4b2775 100644
--- a/tests/test_fileproc.py
+++ b/tests/test_fileproc.py
@@ -2,7 +2,8 @@
 # -*- coding: utf-8 -*-
 import os
 import pytest
-from urlchecker.core.fileproc import check_file_type, get_file_paths, collect_links_from_file, include_file, remove_empty
+import tempfile
+from urlchecker.core.fileproc import check_file_type, get_file_paths, collect_links_from_file, include_file, remove_empty, save_results
 
 
 @pytest.mark.parametrize('file_path', ["tests/test_files/sample_test_file.md",
@@ -83,3 +84,14 @@ def test_remove_empty():
     urls = ["notempty", "notempty", "", None]
     if len(remove_empty(urls)) != 2:
         raise AssertionError
+
+
+def test_save_results():
+    """
+    test that saving results works.
+    """
+    check_results = {"failed": ["fail1", "fail2"], "passed": ["pass1", "pass2"]}
+    output_csv = tempfile.NamedTemporaryFile(suffix=".csv", prefix="urlchecker-").name
+    output_file = save_results(check_results, output_csv)
+    if not os.path.exists(output_csv):
+        raise AssertionError