Repository: cloud-custodian/cloud-custodian

Issue:
windows support redux
Hello

Can anyone provide some advice on the below error when trying to package a simple policy for lambda. 

```
PS C:\Users\USER\Custodian> custodian run --region=eu-west-1 --output-dir=. --config=custod
2017-01-04 10:58:51,770: custodian.policy:INFO Provisioning policy lambda ec2-tag-complianc
2017-01-04 10:58:52,921: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\c7n\policy.py", line 300, in provision
    role=self.policy.options.assume_role)
  File "c:\python27\lib\site-packages\c7n\mu.py", line 221, in publish
    func, role, s3_uri, qualifier=alias)
  File "c:\python27\lib\site-packages\c7n\mu.py", line 319, in _create_or_update
    code_ref = {'ZipFile': archive.get_bytes()}
  File "c:\python27\lib\site-packages\c7n\mu.py", line 165, in get_bytes
    return open(self._temp_archive_file.name, 'rb').read()
IOError: [Errno 13] Permission denied: 'c:\\users\\USER\\appdata\\local\\temp\\tmpvah3za'
2017-01-04 10:58:52,934: custodian.commands:ERROR Error while executing policy ec2-tag-comp
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\c7n\commands.py", line 114, in run
    policy()
  File "c:\python27\lib\site-packages\c7n\policy.py", line 461, in __call__
    return mode.provision()
  File "c:\python27\lib\site-packages\c7n\policy.py", line 300, in provision
    role=self.policy.options.assume_role)
  File "c:\python27\lib\site-packages\c7n\mu.py", line 221, in publish
    func, role, s3_uri, qualifier=alias)
  File "c:\python27\lib\site-packages\c7n\mu.py", line 319, in _create_or_update
    code_ref = {'ZipFile': archive.get_bytes()}
  File "c:\python27\lib\site-packages\c7n\mu.py", line 165, in get_bytes
    return open(self._temp_archive_file.name, 'rb').read()
IOError: [Errno 13] Permission denied: 'c:\\users\\USER\\appdata\\local\\temp\\tmpvah3za'
```

I have checked permissions/ UAC etc


Discussion:
its a permission issue for ops on a closed file, which works on posix but not windows. we had fixed this a while back, but the fix ended up causing a regression that would leak temp dirs, and was reverted.  we should revisit proper support for windows.


References #547 and #118
Would it be easier to shove CC into docker and execute it that way?
probably ;-)  we publish per commit into docker hub, or linux compatibility stuff on windows, but we're pretty close to portable outside of that one semantic around the zip file packaging tempfiles, a context manager at the right level would probably resolve it, tbd.

Got it running on an EC2 instance and getting much further :) 
> we should revisit proper support for windows.

I've had good luck with [Appveyor](https://www.appveyor.com/) for Windows CI. You willing to bite the bullet on complicating our CI story further, @kapilt?
In the Python bug https://bugs.python.org/issue14243 I noticed this recommendation, updating the issue here until I have a chance to play with it...
```
After all, you can currently get deterministic cleanup (with a __del__ fallback) via:

  @contextmanager
  def named_temp(name):
    f = NamedTemporaryFile(name, delete=False)
    try:
        yield f
    finally:
        try:
            os.unlink(name)
        except OSError:
            pass

You need to be careful to make sure you keep the CM alive (or it will delete the file behind your back), but the idiom RDM described in the other issues handles that for you:

  with named_temp(fname) as f:
     data = "Data\n"
     f.write(data)
     f.close() # Windows compatibility
     with open(fname) as f:
         self.assertEqual(f.read(), data)
```
Running into this issue while attempting to test with Windows.
This is still an open issue for running some tests on Windows, most of us developing on Windows are using Windows Subsystem for Linux currently so it hasn't been a blocker - but we should get to it at some point for sure.  It is basically functionality that does not exist in Windows so it will require refactoring a bit.
Sure, I did go down the route of using WSL. If we get to the point where we're deploying automatically we'll have to spin up some Linux hosts since we're largely a Windows shop.

Resolving patch:
diff --git a/c7n/mu.py b/c7n/mu.py
--- a/c7n/mu.py
+++ b/c7n/mu.py
@@ -64,13 +64,20 @@ class PythonPackageArchive(object):
     zip_compression = zipfile.ZIP_DEFLATED
 
     def __init__(self, *modules):
-        self._temp_archive_file = tempfile.NamedTemporaryFile()
+        self._temp_archive_file = tempfile.NamedTemporaryFile(delete=False)
         self._zip_file = zipfile.ZipFile(
             self._temp_archive_file, mode='w',
             compression=self.zip_compression)
         self._closed = False
         self.add_modules(None, *modules)
 
+    def __del__(self):
+        if not self._closed:
+            self.close()
+        if self._temp_archive_file:
+            self._temp_archive_file.close()
+            os.unlink(self.path)
+
     @property
     def path(self):
         return self._temp_archive_file.name
@@ -99,7 +106,7 @@ def add_modules(self, ignore, *modules):
                     # Likely a namespace package. Try to add *.pth files so
                     # submodules are importable under Python 2.7.
 
-                    sitedir = list(module.__path__)[0].rsplit('/', 1)[0]
+                    sitedir = os.path.abspath(os.path.join(list(module.__path__)[0], os.pardir))
                     for filename in os.listdir(sitedir):
                         s = filename.startswith
                         e = filename.endswith
@@ -187,6 +194,8 @@ def add_contents(self, dest, contents):
             dest = zinfo(dest)  # see for some caveats
         # Ensure we apply the compression
         dest.compress_type = self.zip_compression
+        # Mark host OS as Linux for all archives
+        dest.create_system = 3
         self._zip_file.writestr(dest, contents)
 
     def close(self):
@@ -202,10 +211,6 @@ def close(self):
                 1024.0 * 1024.0)))
         return self
 
-    @staticmethod
-    def _temporary_opener(name, flag, mode=0o777):
-        return os.open(name, flag | os.O_TEMPORARY, mode)
-
     def remove(self):
         """Dispose of the temp file for garbage collection."""
         if self._temp_archive_file:
@@ -225,11 +230,7 @@ def get_bytes(self):
     def get_stream(self):
         """Return the entire zip file as a stream. """
         assert self._closed, "Archive not closed"
-        # Windows requires TEMPORARY flag if you want to open files created by tempfile library
-        if os.name == 'nt':
-            return open(self._temp_archive_file.name, 'rb', opener=self._temporary_opener)
-        else:
-            return open(self._temp_archive_file.name, 'rb')
+        return open(self._temp_archive_file.name, 'rb')
 
     def get_reader(self):
         """Return a read-only :py:class:`~zipfile.ZipFile`."""
diff --git a/c7n/policy.py b/c7n/policy.py
--- a/c7n/policy.py
+++ b/c7n/policy.py
@@ -983,7 +983,10 @@ def validate_policy_start_stop(self):
             except Exception as e:
                 raise ValueError(
                     "Policy: %s TZ not parsable: %s, %s" % (policy_name, policy_tz, e))
-            if not isinstance(p_tz, tzutil.tzfile):
+
+            # Type will be tzwin on windows, but tzwin is null on linux
+            if not (isinstance(p_tz, tzutil.tzfile) or
+                    (tzutil.tzwin and isinstance(p_tz, tzutil.tzwin))):
                 raise ValueError(
                     "Policy: %s TZ not parsable: %s" % (policy_name, policy_tz))
 
diff --git a/c7n/resources/s3.py b/c7n/resources/s3.py
--- a/c7n/resources/s3.py
+++ b/c7n/resources/s3.py
@@ -1818,7 +1818,7 @@ def process(self, buckets):
              "remediated:%d rate:%0.2f/s time:%0.2fs"),
             object_count,
             remediated_count,
-            float(object_count) / run_time,
+            float(object_count) / run_time if run_time else 0,
             run_time)
         return results
 
@@ -2562,7 +2562,7 @@ def empty_buckets(self, buckets):
         log.info(
             "EmptyBucket buckets:%d Complete keys:%d rate:%0.2f/s time:%0.2fs",
             len(buckets), object_count,
-            float(object_count) / run_time, run_time)
+            float(object_count) / run_time if run_time else 0, run_time)
         return results
 
     def process_chunk(self, batch, bucket):


Test patch:
diff --git a/c7n/testing.py b/c7n/testing.py
--- a/c7n/testing.py
+++ b/c7n/testing.py
@@ -65,13 +65,14 @@ def write_policy_file(self, policy, format="yaml"):
         Input a dictionary and a format. Valid formats are `yaml` and `json`
         Returns the file path.
         """
-        fh = tempfile.NamedTemporaryFile(mode="w+b", suffix="." + format)
+        fh = tempfile.NamedTemporaryFile(mode="w+b", suffix="." + format, delete=False)
         if format == "json":
             fh.write(json.dumps(policy).encode("utf8"))
         else:
             fh.write(yaml.dump(policy, encoding="utf8", Dumper=yaml.SafeDumper))
 
         fh.flush()
+        self.addCleanup(os.unlink, fh.name)
         self.addCleanup(fh.close)
         return fh.name
 
diff --git a/docs/source/developer/tests.rst b/docs/source/developer/tests.rst
--- a/docs/source/developer/tests.rst
+++ b/docs/source/developer/tests.rst
@@ -18,6 +18,22 @@ Linting can be run with:
 
   $ make lint
 
+Operating System Compatibility
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Tests are currently executed on both Ubuntu 1604 and Windows Server 2019
+and must pass on both operating systems.
+
+Both Windows and Linux sample dockerfiles are provided for running Tox which may help you.
+You can find these in `tools/dev`.
+
+In Docker for Windows you can run both of these containers,
+`even simultaneously <https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/linux-containers>`_.
+
+
+If you need access to Windows you can download a
+`virtual machine <https://developer.microsoft.com/en-us/windows/downloads/virtual-machines>`_
+directly from Microsoft for any hypervisor.
 
 Decorating tests
 ~~~~~~~~~~~~~~~~
diff --git a/tests/test_cache.py b/tests/test_cache.py
--- a/tests/test_cache.py
+++ b/tests/test_cache.py
@@ -19,6 +19,7 @@
 from six.moves import cPickle as pickle
 import tempfile
 import mock
+import os
 
 
 class TestCache(TestCase):
@@ -62,8 +63,7 @@ def setUp(self):
         self.test_value = [1, 2, 3]
 
     def test_get_set(self):
-        t = tempfile.NamedTemporaryFile()
-        self.addCleanup(t.close)
+        t = self.temporary_file_with_cleanup()
         c = cache.FileCacheManager(Namespace(cache_period=60, cache=t.name))
         self.assertFalse(c.load())
         k1 = {"account": "12345678901234", "region": "us-west-2", "resource": "ec2"}
@@ -91,7 +91,8 @@ def test_get(self):
         self.assertEqual(self.test_cache.get(self.bad_key), None)
 
     def test_load(self):
-        t = tempfile.NamedTemporaryFile(suffix=".cache")
+        t = self.temporary_file_with_cleanup(suffix=".cache")
+
         load_config = Namespace(cache_period=0, cache=t.name)
         load_cache = cache.FileCacheManager(load_config)
         self.assertFalse(load_cache.load())
@@ -106,7 +107,8 @@ def test_save_exists(self, mock_dumps, mock_dump, mock_exists, mock_mkdir):
         # path exists then we dont need to create the folder
         mock_exists.return_value = True
         # tempfile to hold the pickle
-        temp_cache_file = tempfile.NamedTemporaryFile()
+        temp_cache_file = self.temporary_file_with_cleanup()
+
         self.test_cache.cache_path = temp_cache_file.name
         # make the call
         self.test_cache.save(self.test_key, self.test_value)
@@ -126,7 +128,8 @@ def test_save_exists(self, mock_dumps, mock_dump, mock_exists, mock_mkdir):
     @mock.patch.object(cache.pickle, "dump")
     @mock.patch.object(cache.pickle, "dumps")
     def test_save_doesnt_exists(self, mock_dumps, mock_dump, mock_exists, mock_mkdir):
-        temp_cache_file = tempfile.NamedTemporaryFile()
+        temp_cache_file = self.temporary_file_with_cleanup()
+
         self.test_cache.cache_path = temp_cache_file.name
 
         # path doesnt exists then we will create the folder
@@ -147,3 +150,16 @@ def test_save_doesnt_exists(self, mock_dumps, mock_dump, mock_exists, mock_mkdir
         self.assertEqual(mock_mkdir.call_count, 1)
         self.assertEqual(mock_dump.call_count, 1)
         self.assertEqual(mock_dumps.call_count, 1)
+
+    def temporary_file_with_cleanup(self, **kwargs):
+        """
+        NamedTemporaryFile with delete=True has
+        significantly different behavior on Windows
+        so we utilize delete=False to simplify maintaining
+        compatibility.
+        """
+        t = tempfile.NamedTemporaryFile(delete=False, **kwargs)
+
+        self.addCleanup(os.unlink, t.name)
+        self.addCleanup(t.close)
+        return t
diff --git a/tests/test_cli.py b/tests/test_cli.py
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -102,6 +102,9 @@ def test_version(self):
     def test_debug_version(self):
         output = self.get_output(["custodian", "version", "--debug"])
         # Among other things, this should print sys.path
+        # Normalize double escaped backslashes for Windows
+        output = output.replace('\\\\', '\\')
+
         self.assertIn(version.version, output)
         self.assertIn(sys.path[0], output)
 
diff --git a/tests/test_mu.py b/tests/test_mu.py
--- a/tests/test_mu.py
+++ b/tests/test_mu.py
@@ -260,7 +260,7 @@ def test_sqs_subscriber(self):
         if self.recording:
             time.sleep(60)
 
-        log_events = list(manager.logs(func, "1970-1-1 UTC", "9170-1-1"))
+        log_events = list(manager.logs(func, "1970-1-1 UTC", "2037-1-1"))
         messages = [
             e["message"] for e in log_events if e["message"].startswith('{"Records')
         ]
@@ -300,7 +300,7 @@ def test_sns_subscriber_and_ipaddress(self):
         client.publish(TopicArn=topic_arn, Message="Greetings, program!")
         if self.recording:
             time.sleep(30)
-        log_events = manager.logs(func, "1970-1-1 UTC", "9170-1-1")
+        log_events = manager.logs(func, "1970-1-1 UTC", "2037-1-1")
         messages = [
             e["message"] for e in log_events if e["message"].startswith('{"Records')
         ]
diff --git a/tests/test_notify.py b/tests/test_notify.py
--- a/tests/test_notify.py
+++ b/tests/test_notify.py
@@ -16,6 +16,7 @@
 from .common import BaseTest, functional
 
 import base64
+import os
 import json
 import time
 import tempfile
@@ -38,11 +39,15 @@ def cleanup():
                 time.sleep(60)
 
         self.addCleanup(cleanup)
-        temp_file = tempfile.NamedTemporaryFile(mode="w")
+        temp_file = tempfile.NamedTemporaryFile(mode="w", delete=False)
         json.dump({"emails": ["me@example.com"]}, temp_file)
         temp_file.flush()
+
+        self.addCleanup(os.unlink, temp_file.name)
         self.addCleanup(temp_file.close)
 
+        url = "file:///%s" % temp_file.name.replace('\\', '/')
+
         policy = self.load_policy(
             {
                 "name": "notify-address",
@@ -53,12 +58,12 @@ def cleanup():
                         "type": "notify",
                         "to": ["to@example.com"],
                         "to_from": {
-                            "url": "file://%s" % temp_file.name,
+                            "url": url,
                             "format": "json",
                             "expr": "emails",
                         },
                         "cc_from": {
-                            "url": "file://%s" % temp_file.name,
+                            "url": url,
                             "format": "json",
                             "expr": "emails",
                         },
diff --git a/tests/test_resolver.py b/tests/test_resolver.py
--- a/tests/test_resolver.py
+++ b/tests/test_resolver.py
@@ -77,7 +77,8 @@ def test_resolve_file(self):
         content = json.dumps({"universe": {"galaxy": {"system": "sun"}}})
         cache = FakeCache()
         resolver = URIResolver(None, cache)
-        with tempfile.NamedTemporaryFile(mode="w+", dir=os.getcwd()) as fh:
+        with tempfile.NamedTemporaryFile(mode="w+", dir=os.getcwd(), delete=False) as fh:
+            self.addCleanup(os.unlink, fh.name)
             fh.write(content)
             fh.flush()
             self.assertEqual(resolver.resolve("file:%s" % fh.name), content)
@@ -114,7 +115,7 @@ def test_txt(self):
         with open("resolver_test.txt", "w") as out:
             for i in ["a", "b", "c", "d"]:
                 out.write("%s\n" % i)
-        with open("resolver_test.txt", "r") as out:
+        with open("resolver_test.txt", "rb") as out:
             values = self.get_values_from({"url": "letters.txt"}, out.read())
         os.remove("resolver_test.txt")
         self.assertEqual(values.get_values(), ["a", "b", "c", "d"])
@@ -123,7 +124,7 @@ def test_csv_expr(self):
         with open("test_expr.csv", "w") as out:
             writer = csv.writer(out)
             writer.writerows([range(5) for r in range(5)])
-        with open("test_expr.csv", "r") as out:
+        with open("test_expr.csv", "rb") as out:
             values = self.get_values_from(
                 {"url": "sun.csv", "expr": "[*][2]"}, out.read()
             )
@@ -135,7 +136,7 @@ def test_csv_expr_using_dict(self):
             writer = csv.writer(out)
             writer.writerow(["aa", "bb", "cc", "dd", "ee"])  # header row
             writer.writerows([range(5) for r in range(5)])
-        with open("test_dict.csv", "r") as out:
+        with open("test_dict.csv", "rb") as out:
             values = self.get_values_from(
                 {"url": "sun.csv", "expr": "bb[1]", "format": "csv2dict"}, out.read()
             )
@@ -146,7 +147,7 @@ def test_csv_column(self):
         with open("test_column.csv", "w") as out:
             writer = csv.writer(out)
             writer.writerows([range(5) for r in range(5)])
-        with open("test_column.csv", "r") as out:
+        with open("test_column.csv", "rb") as out:
             values = self.get_values_from({"url": "sun.csv", "expr": 1}, out.read())
         os.remove("test_column.csv")
         self.assertEqual(values.get_values(), ["1", "1", "1", "1", "1"])
@@ -155,7 +156,7 @@ def test_csv_raw(self):
         with open("test_raw.csv", "w") as out:
             writer = csv.writer(out)
             writer.writerows([range(3, 4) for r in range(5)])
-        with open("test_raw.csv", "r") as out:
+        with open("test_raw.csv", "rb") as out:
             values = self.get_values_from({"url": "sun.csv"}, out.read())
         os.remove("test_raw.csv")
         self.assertEqual(values.get_values(), [["3"], ["3"], ["3"], ["3"], ["3"]])
diff --git a/tests/test_s3.py b/tests/test_s3.py
--- a/tests/test_s3.py
+++ b/tests/test_s3.py
@@ -17,6 +17,7 @@
 import functools
 import json
 import os
+import io
 import shutil
 import tempfile
 import time  # NOQA needed for some recordings
@@ -1274,7 +1275,7 @@ def tell(self):
             Bucket=bname,
             Key=key,
             Metadata={"planet": "earth"},
-            Body=wrapper(open("/dev/zero"), size),
+            Body=wrapper(io.BytesIO(bytearray(size)), size),
             ContentLength=size,
         )
         info = client.head_object(Bucket=bname, Key=key)
diff --git a/tests/zpill.py b/tests/zpill.py
--- a/tests/zpill.py
+++ b/tests/zpill.py
@@ -234,6 +234,7 @@ def get_next_file_path(self, service, operation):
         while next_file is None:
             index = self._index.setdefault(base_name, 1)
             fn = os.path.join(self._data_path, base_name + "_{0}.json".format(index))
+            fn = fn.replace('\\', '/')
             if fn in self._files:
                 next_file = fn
                 self._index[base_name] += 1
diff --git a/tools/c7n_mailer/tests/test_email.py b/tools/c7n_mailer/tests/test_email.py
--- a/tools/c7n_mailer/tests/test_email.py
+++ b/tools/c7n_mailer/tests/test_email.py
@@ -54,6 +54,10 @@ def setUp(self):
         self.email_delivery.ldap_lookup.uid_regex = ''
         template_abs_filename = os.path.join(os.path.abspath(os.path.dirname(__file__)),
                                              'example.jinja')
+
+        # Jinja paths must always be forward slashes regardless of operating system
+        template_abs_filename = template_abs_filename.replace('\\', '/')
+
         SQS_MESSAGE_1['action']['template'] = template_abs_filename
         SQS_MESSAGE_4['action']['template'] = template_abs_filename