Repository: python-pillow/Pillow

Issue:
Use unittest for tests
Replaces https://github.com/python-pillow/Pillow/pull/693 and preserves history.

Notes:
- The old `tester.py` is now `helper.py` and contains helper functions like `lena()` and a base class `PillowTestCase(unittest.TestCase)`.
- unittest.TestCase has many [assert functions](https://docs.python.org/2/library/unittest.html#assert-methods) for Python 2.7 onwards. For Python 2.6 we use `unittest2` which has them backported.
- `PillowTestCase` defines the extra asserts like `assert_image_equal()`.
- Individual `test/test_*.py` files have a test class that derives from `PillowTestCase` so they can use `assert_image_equal()` and so on.
- Individual `test/test_*.py` have `if __name__ == '__main__': unittest.main()` so they can be individually run: `python test/test_something.py`
- To run the whole batch (e.g. on Travis CI) it uses `nosetests` (see `.travis.yml` for latest command): 
  - No coverage: `nosetests Tests/test_*.py`
  - With coverage: `coverage run --append --include=PIL/* -m nose Tests/test_*.py`

TODO:
- [x] `helper.py` still has some commented out stuff from `tester.py`.
- [ ] The old test runner prints out remaining temp files at the end. This new one does so when running an individual test file, but not via nose.
- [ ] `lena()` opens images fresh from disk to ensure tests are independent and repeatable. It would be better to caches images for speed (no caching is around 60% slower). What's the best way to create a 100% duplicate copy of an image? There's no point testing on a copy if it loses important information from the original.
- [ ] Send `test_olefileio.py` back upstream.
- [ ] There's some miscellaneous files still in `Tests/`. Are these still needed?
  - `Tests/bench_cffi_access.py`
  - `Tests/bench_get.py`
  - `Tests/cms_test.py`
  - `Tests/crash_ttf_memory_error.py`
  - `Tests/helper.py`
  - `Tests/import_all.py`
  - `Tests/large_memory_numpy_test.py`
  - `Tests/large_memory_test.py`
  - `Tests/make_hash.py`
  - `Tests/show_icc.py`
  - `Tests/show_mcidas.py`
  - `Tests/threaded_save.py`
  - `Tests/versions.py`



Discussion:
[![Coverage Status](https://coveralls.io/builds/852626/badge)](https://coveralls.io/builds/852626)

Coverage decreased (-0.48%) when pulling **51427436d2e480dac6dae6cb607256c6fc9df1f4 on hugovk:unittest3** into **6a79d80374e0e1c14601c4e12724e4507d4cd834 on python-pillow:master**.

Coverage decreased because more code is included in the report.

Before: 6584 OF 9036 RELEVANT LINES COVERED
After: 6587 OF 9100 RELEVANT LINES COVERED

These two files weren't included before and they brought the average down a bit:

```
COVERAGE        FILE    LINES   RELEVANT    COVERED MISSED  HITS/LINE
25.0%       PIL/ImageGrab.py    49  20  5   15  1.0
13.64%      PIL/ImageQt.py  89  44  6   38  1.0
```

Before: https://coveralls.io/builds/852429
After: https://coveralls.io/builds/852673

Yeah, I'm just happy we are tracking coverage :+1:

Re the other tests:
- make_hash.py is used in the code (storage.c?) somewhere to determine some hard coded hash constants. We'd need to run that if there are changes that cause a hash collision.
- large_* are tests that are potential DOS against the test machine. Worth keeping, and worth running  explicitly before releases, but not all the time. 
- bench_cffi_access,py is a benchmark. Again, it's a DOS against the test machine. 

These at least are useful. 

Ok, this is still somewhat troublesome OMM, or at least for how I'm working.

&lt;parenthetical> 

For clarity, my setup is:

A bunch of virtualenvs:

```
~/vpy26
~/vpy27
~/vpy27-dbg
~/vpy32
~/vpy33
~/vpy33-dbg
~/vpy34
~/vpypy
```

Plus one Pillow directory. From within the pillow directory, I hack, and then:

```
source ~vpy27/bin/activate
python setup.py install && python Tests/run.py --installed
[[ repeat ]]
```

And in another terminal window, I'll have a 3.x virtualenv going with the same set of commands. If necessary, I'll use the dbg versions for gdb, or whatever. 

&lt;/parenthetical>

The root problem, I think, is that these tests are working against an in place build, not an installed version. We worked around the import issue in the old test suite with the `--installed` flag.

I don't like in place builds for a few reasons:
- It's harder to do a clean build. The output files are intermingled with the source files. With non-in place builds, it's just `rm -f build` and I get a clean build across all of my python versions. (or `python setup.py clean` for just that arch. )
- You can't have Pillow for two different python versions installed at once.  This is very useful when I'm tracking down weird crashing bugs. 
- It's not immediately obvious which version the .so files are compiled for. You just get strange import errors. 
- `python setup.py develop` was a useful hack pre-virtualenv, It's less useful now that we can have full virtualenvs that don't require sudo for install. 

I also think that it's extremely valuable to be able to test the current installed Pillow build without building whatever is currently in the source directory. 

@wiredfool Can we support both in-place and installed test running? For whatever it's worth, I think I find installed-test-running less attractive.

I'm not arguing for removing in place -- Clearly others find it useful. I'm arguing for whatever we do retaining the ability to test installed code. 

:+1:

@hugovk Can that be added or do we need to revert again? :smile:

OK, so the [old runner calls](https://github.com/python-pillow/Pillow/blob/master/Tests/run.py#L27-L28) `os.environ["PYTHONPATH"] = "."` early on when `--installed` is set.

@wiredfool Would an environment variable be ok rather than a command-line argument? That would be easier to implement for nose, otherwise we'll probably need different handling to run the full set with nose (perhaps via a [plugin like this](http://www.siafoo.net/article/54)) and individually. I'd like to keep it so the tests can be run individually without nose (`python Tests/test_file.py`), and altogether in a batch with nose. 

(Likewise there should be no problem running individual tests with nose (`nosetests Tests/test_file.py`) or the whole set without (something like `python -m unittest discover -p 'Tests/test_*.py`).)

It looks like the old runner calls `os.environ["PYTHONPATH"] = "."` early on when `--installed` is **not** set.

An environment variable is ok, not what I'd prefer, but it's going to be easier than monkeypatching. (the difference between FOO=bar nosetests and nosetests--foo-bar is minimal, just different typing)

There are a couple of possible command line flags for nosetests, --first-package-wins and --no-path-adjustment that might do something useful. 

I could swear that there was some monkeying around with sys.path to make the imports work in the old testing setup, but it's just not there. (Well, it's in `selftest.py`, maybe that's what I was thinking.) 

`nosetests -s --no-path-adjustment` gets the imports of PIL right, but it falls down on importing helper. Relative imports don't help, since we aren't importing the Tests/ module. 

e.g., this test file:

```
import sys
print (sys.path)
import PIL
print (PIL.__file__)
from helper import unittest, PillowTestCase, tearDownModule, lena, py3
```

yields this:

```
$ nosetests --no-path-adjustment -s Tests/test_nose.py
['/home/erics/vpy33/bin', '/home/erics/vpy33/lib/python3.3/site-packages/distribute-0.6.34-py3.3.egg', '/home/erics/vpy33/lib/python3.3/site-packages/pip-1.3.1-py3.3.egg', '/home/erics/vpy33/lib/python3.3/site-packages/Pillow-2.4.0-py3.3-linux-x86_64.egg', '/home/erics/vpy33/lib/python3.3', '/home/erics/vpy33/lib/python3.3/plat-linux', '/home/erics/vpy33/lib/python3.3/lib-dynload', '/usr/lib/python3.3', '/usr/lib/python3.3/plat-linux', '/home/erics/vpy33/lib/python3.3/site-packages']
/home/erics/vpy33/lib/python3.3/site-packages/Pillow-2.4.0-py3.3-linux-x86_64.egg/PIL/__init__.py
E
======================================================================
ERROR: Failure: ImportError (No module named 'helper')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/erics/vpy33/lib/python3.3/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/erics/vpy33/lib/python3.3/site-packages/nose/loader.py", line 414, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/erics/vpy33/lib/python3.3/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/erics/vpy33/lib/python3.3/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/erics/vpy33/lib/python3.3/imp.py", line 180, in load_module
    return load_source(name, filename, file)
  File "/home/erics/vpy33/lib/python3.3/imp.py", line 119, in load_source
    _LoadSourceCompatibility(name, pathname, file).load_module(name)
  File "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1022, in load_module
  File "<frozen importlib._bootstrap>", line 1003, in load_module
  File "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 868, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "/home/erics/Pillow/Tests/test_nose.py", line 6, in <module>
    from helper import unittest, PillowTestCase, tearDownModule, lena, py3
ImportError: No module named 'helper'
```

And also this:

```
(vpy33)erics@builder-1204-x64:~/Pillow$ nosetests -s Tests/test_nose.py
['/home/erics/Pillow/Tests', '/home/erics/Pillow', '/home/erics/vpy33/bin', '/home/erics/vpy33/lib/python3.3/site-packages/distribute-0.6.34-py3.3.egg', '/home/erics/vpy33/lib/python3.3/site-packages/pip-1.3.1-py3.3.egg', '/home/erics/vpy33/lib/python3.3/site-packages/Pillow-2.4.0-py3.3-linux-x86_64.egg', '/home/erics/vpy33/lib/python3.3', '/home/erics/vpy33/lib/python3.3/plat-linux', '/home/erics/vpy33/lib/python3.3/lib-dynload', '/usr/lib/python3.3', '/usr/lib/python3.3/plat-linux', '/home/erics/vpy33/lib/python3.3/site-packages']
/home/erics/Pillow/PIL/__init__.py

----------------------------------------------------------------------
Ran 0 tests in 0.001s

OK
```

So, nose, by default adds the absolute path for .  and the test location to the python path. I/We need the test location, but not the initial directory. 

So far, I haven't been able to get it to work on command lines or from various options to `nose.run()`

No luck so far. 

I found one suggestion saying to create a blank `Tests/__init__.py` file. This worked locally for me from a 2.x virtualenv and 2.x on Travis but [not for 3.x on Travis](https://travis-ci.org/hugovk/Pillow/jobs/27301768).

I'm on Windows right now but I had problems on Mac with `python setup.py install` from a 3.4 virtualenv. It falls over with TypeError("Can't mix strings and bytes in path components.") from posixpath.py.


Resolving patch:
diff --git a/test-installed.py b/test-installed.py
new file mode 100755
--- /dev/null
+++ b/test-installed.py
@@ -0,0 +1,24 @@
+#!/usr/bin/env python
+import nose
+import os
+import sys
+import glob
+
+# monkey with the path, removing the local directory but adding the Tests/ directory
+# for helper.py and the other local imports there.
+
+del(sys.path[0])
+sys.path.insert(0, os.path.abspath('./Tests'))
+
+# if there's no test selected (mostly) choose a working default.
+# Something is required, because if we import the tests from the local
+# directory, once again, we've got the non-installed PIL in the way
+if len(sys.argv) == 1:
+    sys.argv.extend(glob.glob('Tests/test*.py'))
+
+# Make sure that nose doesn't muck with our paths. 
+if ('--no-path-adjustment' not in sys.argv) and ('-P' not in sys.argv):
+    sys.argv.insert(1, '--no-path-adjustment')
+
+if __name__ == '__main__':
+    nose.main()


Test patch:
diff --git a/Tests/test_imagecms.py b/Tests/test_imagecms.py
--- a/Tests/test_imagecms.py
+++ b/Tests/test_imagecms.py
@@ -18,6 +18,8 @@ class TestImageCms(PillowTestCase):
     def setUp(self):
         try:
             from PIL import ImageCms
+            # need to hit getattr to trigger the delayed import error
+            ImageCms.core.profile_open
         except ImportError as v:
             self.skipTest(v)