Repository: spotify/luigi

Issue:
range_test and sge_test not picked up by tox due to being executable
While developing some new functionality in `luigi.tools.range` I ran the `range_test.py` file manually and noticed that it had multiple failing tests that were unrelated to my changes, even on a clean master with no changes at all. Strangely, tox goes through and the build passes anyway. This happens because _the whole file is ignored_. The reason the file is ignored is because it is set as executable (755) which apparently causes tox to ignore it.

**Steps to reproduce**

1. Run complete test suite as usual and watch everything go through: `tox`
2. Run only the `range_test.py` file and it has multiple issues, FAILED (failures=2, errors=6): `cd test && python -m unittest range_test`
3. Change file permissions: `chmod 644 range_test.py`
4. Run tox suite again and this time the file is picked up, failing with the same problems:

```
======================================================================
ERROR: Verify that you can still programmatically set of param as string
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lena/projects/github/spotify/luigi/test/range_test.py", line 1385, in test_old_month_instantiation
    self.assertEqual(expected_task, list(range_task._requires())[0])
IndexError: list index out of range

======================================================================
ERROR: test_param_name (range_test.MonthInstantiationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lena/projects/github/spotify/luigi/test/range_test.py", line 1423, in test_param_name
    self.assertEqual(expected_task, list(range_task._requires())[0])
IndexError: list index out of range

======================================================================
ERROR: test_param_name_with_inferred_fs (range_test.MonthInstantiationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lena/projects/github/spotify/luigi/test/range_test.py", line 1439, in test_param_name_with_inferred_fs
    self.assertEqual(expected_task, list(range_task._requires())[0])
IndexError: list index out of range

======================================================================
FAIL: Verify that you can still use Range through CLI
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lena/projects/github/spotify/luigi/test/range_test.py", line 1407, in test_month_cli_instantiation
    self.assertEqual(MyTask(month_param=datetime.date(1934, 12, 1)).secret, 'yay')
AssertionError: 'some-value-to-sooth-python-linters' != 'yay'
-------------------- >> begin captured logging << --------------------
luigi: INFO: logging already configured
luigi.scheduler: DEBUG: Starting pruning of task graph
luigi.scheduler: DEBUG: Done pruning task graph
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_of_param_commandline (range_test.MonthInstantiationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lena/projects/github/spotify/luigi/test/range_test.py", line 1481, in test_of_param_commandline
    self.assertEqual(MyTask.state, ('bar', 5))
AssertionError: Tuples differ: (None, None) != ('bar', 5)

First differing element 0:
None
'bar'

- (None, None)
+ ('bar', 5)

======================================================================
ERROR: test_non_dividing_interval (range_test.RangeByMinutesBaseTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lena/projects/github/spotify/luigi/test/range_test.py", line 626, in test_non_dividing_interval
    self.assertRaises(luigi.parameter.ParameterException, task.requires)
  File "/usr/lib/python3.6/unittest/case.py", line 733, in assertRaises
    return context.handle('assertRaises', args, kwargs)
  File "/usr/lib/python3.6/unittest/case.py", line 178, in handle
    callable_obj(*args, **kwargs)
  File "/home/lena/virtualenvs/luigi/lib/python3.6/site-packages/luigi-2.8.7-py3.6.egg/luigi/tools/range.py", line 240, in requires
    self._emit_metrics(missing_datetimes, finite_start, finite_stop)
  File "/home/lena/virtualenvs/luigi/lib/python3.6/site-packages/luigi-2.8.7-py3.6.egg/luigi/tools/range.py", line 177, in _emit_metrics
    delay_in_jobs = len(datetimes) - datetimes.index(missing_datetimes[0]) if datetimes and missing_datetimes else 0
ValueError: datetime.datetime(2016, 3, 31, 0, 0) is not in list

======================================================================
ERROR: test_bulk_complete_correctly_interfaced (range_test.RangeByMinutesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lena/projects/github/spotify/luigi/test/range_test.py", line 1463, in test_bulk_complete_correctly_interfaced
    actual = [str(t) for t in task.requires()]
  File "/home/lena/virtualenvs/luigi/lib/python3.6/site-packages/luigi-2.8.7-py3.6.egg/luigi/tools/range.py", line 233, in requires
    missing_datetimes = sorted(self._missing_datetimes(datetimes))
  File "/home/lena/virtualenvs/luigi/lib/python3.6/site-packages/luigi-2.8.7-py3.6.egg/luigi/tools/range.py", line 272, in _missing_datetimes
    return self.missing_datetimes(finite_datetimes)
  File "/home/lena/virtualenvs/luigi/lib/python3.6/site-packages/luigi-2.8.7-py3.6.egg/luigi/tools/range.py", line 798, in missing_datetimes
    complete_parameters = self.of.bulk_complete.__func__(cls_with_params, map(self.datetime_to_parameter, finite_datetimes))
  File "/home/lena/projects/github/spotify/luigi/test/range_test.py", line 1447, in bulk_complete
    return parameter_tuples[:-2]
TypeError: 'map' object is not subscriptable

======================================================================
ERROR: test_bulk_complete_of_params (range_test.RangeByMinutesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lena/projects/github/spotify/luigi/test/range_test.py", line 1493, in test_bulk_complete_of_params
    actual = [str(t) for t in task.requires()]
  File "/home/lena/virtualenvs/luigi/lib/python3.6/site-packages/luigi-2.8.7-py3.6.egg/luigi/tools/range.py", line 233, in requires
    missing_datetimes = sorted(self._missing_datetimes(datetimes))
  File "/home/lena/virtualenvs/luigi/lib/python3.6/site-packages/luigi-2.8.7-py3.6.egg/luigi/tools/range.py", line 272, in _missing_datetimes
    return self.missing_datetimes(finite_datetimes)
  File "/home/lena/virtualenvs/luigi/lib/python3.6/site-packages/luigi-2.8.7-py3.6.egg/luigi/tools/range.py", line 798, in missing_datetimes
    complete_parameters = self.of.bulk_complete.__func__(cls_with_params, map(self.datetime_to_parameter, finite_datetimes))
  File "/home/lena/projects/github/spotify/luigi/test/range_test.py", line 1476, in bulk_complete
    return parameter_tuples[:-2]
TypeError: 'map' object is not subscriptable
``` 

The only executable files I could find were `range_test.py` and `contrib/sge_test.py`.

**Solution**

We need to chmod the concerned files to match the others (644) and fix the failing tests. If we can't fix them right now we need to temporarily ignore the individual tests and create new Github issues. I am in progress of doing this but wanted to report the error in case anyone else encounters the problem.


Discussion:
@lallea FYI
Thanks @lnsndn for debugging and fixing. Probably my bad.

I reviewed and concluded that the tests are wrong, but the implementation is right. The tests were probably passing before https://github.com/spotify/luigi/pull/2666 got merged. 

Resolving patch:
diff --git a/luigi/contrib/sge.py b/luigi/contrib/sge.py
--- a/luigi/contrib/sge.py
+++ b/luigi/contrib/sge.py
@@ -278,9 +278,11 @@ def _dump(self, out_dir=''):
                 d = pickle.dumps(self)
                 module_name = os.path.basename(sys.argv[0]).rsplit('.', 1)[0]
                 d = d.replace('(c__main__', "(c" + module_name)
-                open(self.job_file, "w").write(d)
+                with open(self.job_file, "w") as f:
+                    f.write(d)
             else:
-                pickle.dump(self, open(self.job_file, "w"))
+                with open(self.job_file, "wb") as f:
+                    pickle.dump(self, f)
 
     def _run_job(self):
 
diff --git a/luigi/tools/range.py b/luigi/tools/range.py
--- a/luigi/tools/range.py
+++ b/luigi/tools/range.py
@@ -466,7 +466,7 @@ def finite_datetimes(self, finite_start, finite_stop):
         # Validate that the minutes_interval can divide 60 and it is greater than 0 and lesser than 60
         if not (0 < self.minutes_interval < 60):
             raise ParameterException('minutes-interval must be within 0..60')
-        if (60 / self.minutes_interval) * self.minutes_interval != 60:
+        if 60 % self.minutes_interval != 0:
             raise ParameterException('minutes-interval does not evenly divide 60')
         # start of a complete interval, e.g. 20:13 and the interval is 5 -> 20:10
         start_minute = int(finite_start.minute/self.minutes_interval)*self.minutes_interval


Test patch:
diff --git a/test/contrib/sge_test.py b/test/contrib/sge_test.py
old mode 100755
new mode 100644
--- a/test/contrib/sge_test.py
+++ b/test/contrib/sge_test.py
@@ -53,7 +53,7 @@ def on_sge_master():
 class TestSGEWrappers(unittest.TestCase):
 
     def test_track_job(self):
-        '''`track_job` returns the state using qstat'''
+        """`track_job` returns the state using qstat"""
         self.assertEqual(_parse_qstat_state(QSTAT_OUTPUT, 1), 'r')
         self.assertEqual(_parse_qstat_state(QSTAT_OUTPUT, 2), 'qw')
         self.assertEqual(_parse_qstat_state(QSTAT_OUTPUT, 3), 't')
@@ -63,7 +63,7 @@ def test_track_job(self):
 
 class TestJobTask(SGEJobTask):
 
-    '''Simple SGE job: write a test file to NSF shared drive and waits a minute'''
+    """Simple SGE job: write a test file to NSF shared drive and waits a minute"""
 
     i = luigi.Parameter()
 
@@ -79,7 +79,7 @@ def output(self):
 @attr('contrib')
 class TestSGEJob(unittest.TestCase):
 
-    '''Test from SGE master node'''
+    """Test from SGE master node"""
 
     def test_run_job(self):
         if on_sge_master():
@@ -94,7 +94,7 @@ def test_run_job_with_dump(self, mock_check_output):
             'Your job 12345 ("test_job") has been submitted',
             ''
         ]
-        task = TestJobTask(i=1, n_cpu=1, shared_tmp_dir='/tmp')
+        task = TestJobTask(i="1", n_cpu=1, shared_tmp_dir='/tmp')
         luigi.build([task], local_scheduler=True)
         self.assertEqual(mock_check_output.call_count, 2)
 
diff --git a/test/range_test.py b/test/range_test.py
old mode 100755
new mode 100644
--- a/test/range_test.py
+++ b/test/range_test.py
@@ -1100,7 +1100,7 @@ class MyTask(luigi.Task):
             def complete(self):
                 return False
 
-        range_task = RangeMonthly(now=datetime_to_epoch(datetime.datetime(2015, 12, 2)),
+        range_task = RangeMonthly(now=datetime_to_epoch(datetime.datetime(2016, 1, 1)),
                                   of=MyTask,
                                   start=datetime.date(2015, 12, 1),
                                   stop=datetime.date(2016, 1, 1))
@@ -1125,7 +1125,7 @@ def run(self):
                 self.comp = True
                 MyTask.secret = 'yay'
 
-        now = str(int(datetime_to_epoch(datetime.datetime(2015, 12, 2))))
+        now = str(int(datetime_to_epoch(datetime.datetime(2016, 1, 1))))
         self.run_locally_split('RangeMonthly --of wohoo.MyTask --now {now} --start 2015-12 --stop 2016-01'.format(now=now))
         self.assertEqual(MyTask(month_param=datetime.date(1934, 12, 1)).secret, 'yay')
 
@@ -1137,7 +1137,7 @@ class MyTask(luigi.Task):
             def complete(self):
                 return False
 
-        range_task = RangeMonthly(now=datetime_to_epoch(datetime.datetime(2015, 12, 2)),
+        range_task = RangeMonthly(now=datetime_to_epoch(datetime.datetime(2016, 1, 1)),
                                   of=MyTask,
                                   start=datetime.date(2015, 12, 1),
                                   stop=datetime.date(2016, 1, 1),
@@ -1153,7 +1153,7 @@ class MyTask(luigi.Task):
             def output(self):
                 return MockTarget(self.month_param.strftime('/n2000y01a05n/%Y_%m-aww/21mm%Hdara21/ooo'))
 
-        range_task = RangeMonthly(now=datetime_to_epoch(datetime.datetime(2015, 12, 2)),
+        range_task = RangeMonthly(now=datetime_to_epoch(datetime.datetime(2016, 1, 1)),
                                   of=MyTask,
                                   start=datetime.date(2015, 12, 1),
                                   stop=datetime.date(2016, 1, 1),
@@ -1197,7 +1197,7 @@ def run(self):
                 self.comp = True
                 MyTask.state = (self.arbitrary_param, self.arbitrary_integer_param)
 
-        now = str(int(datetime_to_epoch(datetime.datetime(2015, 12, 2))))
+        now = str(int(datetime_to_epoch(datetime.datetime(2016, 1, 1))))
         self.run_locally(['RangeMonthly', '--of', 'wohoo.MyTask',
                           '--of-params', '{"arbitrary_param":"bar","arbitrary_integer_param":5}',
                           '--now', '{0}'.format(now), '--start', '2015-12', '--stop', '2016-01'])
@@ -1444,7 +1444,7 @@ class BulkCompleteByMinutesTask(luigi.Task):
 
             @classmethod
             def bulk_complete(cls, parameter_tuples):
-                return parameter_tuples[:-2]
+                return list(parameter_tuples)[:-2]
 
             def output(self):
                 raise RuntimeError("Shouldn't get called while resolving deps via bulk_complete")
@@ -1471,9 +1471,10 @@ class BulkCompleteByMinutesTask(luigi.Task):
 
             @classmethod
             def bulk_complete(cls, parameter_tuples):
+                ptuples = list(parameter_tuples)
                 for t in map(cls, parameter_tuples):
                     assert t.arbitrary_argument
-                return parameter_tuples[:-2]
+                return ptuples[:-2]
 
             def output(self):
                 raise RuntimeError("Shouldn't get called while resolving deps via bulk_complete")