Repository: certbot/certbot

Issue:
Pin Python dependencies installed in DNS plugins
While #6580 suggests a simple way to pin Python dependencies in our main Docker image, doing it each for each of our DNS plugins is slightly more work but also valuable to avoid the same problem.

The way we pin these dependencies elsewhere is by using https://github.com/certbot/certbot/blob/master/tools/dev_constraints.txt which is a catch all list of all dependencies. Maybe we use this catch all file that we initially made just for our dev setup, but I'm not sure it will work because it's not in the build context of the Dockerfile which is its directory and everything it contains. Maybe we can change the build context on Docker Hub? Even so, using this dev file feels a little hacky to me.

Another way we could approach is this is creating new files in each DNS plugin directory pinning the dependencies it has. These dependencies can be found by running `pip freeze` and I think that all of our packages and dependencies already pinned in #6580 should be removed these files.

While this works, I'd like to avoid us having multiple sets of dependencies we have to remember to update so after doing this, I think we should also rework the scripts in `tools` to use these new files as well. https://github.com/certbot/certbot/blob/master/tools/merge_requirements.py could be used to merge this into a single file and handle shared dependencies.


Discussion:

Resolving patch:
diff --git a/tools/merge_requirements.py b/tools/merge_requirements.py
--- a/tools/merge_requirements.py
+++ b/tools/merge_requirements.py
@@ -10,27 +10,36 @@
 import sys
 
 
-def read_file(file_path):
-    """Reads in a Python requirements file.
+def process_entries(entries):
+    """
     Ignore empty lines, comments and editable requirements
 
-    :param str file_path: path to requirements file
+    :param list entries: List of entries
 
     :returns: mapping from a project to its pinned version
     :rtype: dict
-
     """
     data = {}
-    with open(file_path) as file_h:
-        for line in file_h:
-            line = line.strip()
-            if line and not line.startswith('#') and not line.startswith('-e'):
-                project, version = line.split('==')
-                if not version:
-                    raise ValueError("Unexpected syntax '{0}'".format(line))
-                data[project] = version
+    for e in entries:
+        e = e.strip()
+        if e and not e.startswith('#') and not e.startswith('-e'):
+            project, version = e.split('==')
+            if not version:
+                raise ValueError("Unexpected syntax '{0}'".format(e))
+            data[project] = version
     return data
 
+def read_file(file_path):
+    """Reads in a Python requirements file.
+
+    :param str file_path: path to requirements file
+
+    :returns: list of entries in the file
+    :rtype: list
+
+    """
+    with open(file_path) as file_h:
+        return file_h.readlines()
 
 def output_requirements(requirements):
     """Prepare print requirements to stdout.
@@ -46,14 +55,25 @@ def main(*paths):
     """Merges multiple requirements files together and prints the result.
 
     Requirement files specified later in the list take precedence over earlier
-    files.
+    files. Files are read from file paths passed from the command line arguments.
 
-    :param tuple paths: paths to requirements files
+    If no command line arguments are defined, data is read from stdin instead.
+
+    :param tuple paths: paths to requirements files provided on command line
 
     """
     data = {}
-    for path in paths:
-        data.update(read_file(path))
+    if paths:
+        for path in paths:
+            data.update(process_entries(read_file(path)))
+    else:
+        # Need to check if interactive to avoid blocking if nothing is piped
+        if not sys.stdin.isatty():
+            stdin_data = []
+            for line in sys.stdin:
+                stdin_data.append(line)
+            data.update(process_entries(stdin_data))
+
     return output_requirements(data)
 
 
diff --git a/tools/pip_install.py b/tools/pip_install.py
--- a/tools/pip_install.py
+++ b/tools/pip_install.py
@@ -19,6 +19,7 @@
 
 import merge_requirements as merge_module
 import readlink
+import strip_hashes
 
 
 def find_tools_path():
@@ -47,10 +48,8 @@ def certbot_normal_processing(tools_path, test_constraints):
     with open(certbot_requirements, 'r') as fd:
         data = fd.readlines()
     with open(test_constraints, 'w') as fd:
-        for line in data:
-            search = re.search(r'^(\S*==\S*).*$', line)
-            if search:
-                fd.write('{0}{1}'.format(search.group(1), os.linesep))
+        data = "\n".join(strip_hashes.process_entries(data))
+        fd.write(data)
 
 
 def merge_requirements(tools_path, requirements, test_constraints, all_constraints):
diff --git a/tools/strip_hashes.py b/tools/strip_hashes.py
new file mode 100755
--- /dev/null
+++ b/tools/strip_hashes.py
@@ -0,0 +1,50 @@
+#!/usr/bin/env python
+"""Removes hash information from requirement files passed to it as file path
+arguments or simply piped to stdin."""
+
+import re
+import sys
+
+
+def process_entries(entries):
+    """Strips off hash strings from dependencies.
+
+    :param list entries: List of entries
+
+    :returns: list of dependencies without hashes
+    :rtype: list
+    """
+    out_lines = []
+    for e in entries:
+        e = e.strip()
+        search = re.search(r'^(\S*==\S*).*$', e)
+        if search:
+            out_lines.append(search.group(1))
+    return out_lines
+
+def main(*paths):
+    """
+    Reads dependency definitions from a (list of) file(s) provided on the
+    command line. If no command line arguments are present, data is read from
+    stdin instead.
+
+    Hashes are removed from returned entries.
+    """
+
+    deps = []
+    if paths:
+        for path in paths:
+            with open(path) as file_h:
+                deps += process_entries(file_h.readlines())
+    else:
+        # Need to check if interactive to avoid blocking if nothing is piped
+        if not sys.stdin.isatty():
+            stdin_data = []
+            for line in sys.stdin:
+                stdin_data.append(line)
+            deps += process_entries(stdin_data)
+
+    return "\n".join(deps)
+
+if __name__ == '__main__':
+    print(main(*sys.argv[1:]))  # pylint: disable=star-args


Test patch: