Project: datamanagement
Issue DM-6998: Problems with MemoryTest ordering
{{MemoryTestCase}} (or a derivative thereof) must be run as the last of all tests in a module in order to properly catch leaks. [Our documentation|https://developer.lsst.io/coding/python_testing.html#memory-and-file-descriptor-leak-testing] implies, and [SQR-012 states|https://sqr-012.lsst.io/#memory-test], that this can be achieved by listing it as the last test case in the file. This works for py.test, but not when using plain old unittest: the latter does not, so far as I can see, guarantee any sort of ordering as a matter of principle, and, in practice, it sorts things lexicographically (it uses whatever order it gets from running {{dir()}} on the test module, and I don't *think* that's guaranteed to be anything in particular). For example, consider [{{testAstrometrySourceSelector.py}}|https://github.com/lsst/meas_algorithms/blob/master/tests/testAstrometrySourceSelector.py]. I made the following change to introduce a memory leak: {code} --- a/tests/testAstrometrySourceSelector.py +++ b/tests/testAstrometrySourceSelector.py @@ -70,8 +70,9 @@ class TestAstrometrySourceSelector(lsst.utils.tests.TestCase): self.sourceSelector = sourceSelector.sourceSelectorRegistry['astrometry']() def tearDown(self): - del self.src - del self.sourceSelector + pass + #del self.src + #del self.sourceSelector def testSelectSources_good(self): for i in range(5): {code} Py.test catches it: {code} $ py.test-2.7 testAstrometrySourceSelector.py [...] testAstrometrySourceSelector.py .........F [...] {code} But simply running the test suite does not: {code} $ python testAstrometrySourceSelector.py .......... ---------------------------------------------------------------------- Ran 10 tests in 0.105s {code} Rename the test case: {code} @@ -144,7 +145,7 @@ def setup_module(module): lsst.utils.tests.init() -class MyMemoryTestCase(lsst.utils.tests.MemoryTestCase): +class xMyMemoryTestCase(lsst.utils.tests.MemoryTestCase): pass if __name__ == "__main__": {code} And boom: {code} $ python testAstrometrySourceSelector.py ......... 54 Objects leaked: {code} Based on a very quick check, I think [sconsUtils runs tests by simply invoking {{python}}|https://github.com/lsst/sconsUtils/blob/f9763768d999cefa4c26b9f3418c28394dfb38df/python/lsst/sconsUtils/tests.py#L133], and I'm pretty sure that this is hard-wired into the muscle memory of many developers. In these cases, memory tests written following current guidelines won't be being properly executed.
Story points: 3