Repository: vispy/vispy

Issue:
Instructions for testing VisPy canvas on Travis
It would be great to put instructions somewhere detailing how to unit test VisPy visualizations on Travis. Typically, I have a project on GitHub that uses py.test and Travis. One of my tests displays a custom visualization created with VisPy (using `canvas.show(); app.run()`). Running `make test` locally opens a window, which has to be closed manually.

On Travis, I'd like at least the GLIR commands to be emitted, even if they are not interpreted by anything. This will test my data baking logic, for example.

What are the exact steps to make this work correctly on Travis? (putting the steps in this issue's discussion would be enough at first)



Discussion:
And to go a bit further: are there ways to test user interaction? (like click at (300, 400), drag the mouse button 20 px left, etc.)

For visuals testing, take a look at #706. It actually checks the outputs to make sure they are consistent. If your visuals are stable enough to do that, you can use it. It uses a threshold for a correlation (default 0.95), so even if there is a little bit of variability it's considered okay.

Also, all of the examples get tested -- at least that they run without errors or warnings -- by `python make test examples`, so that's another layer of testing we do.

Button presses are faked natively for each Canvas here, which isn't quite what you want:

https://github.com/vispy/vispy/blob/master/vispy/app/tests/test_app.py

Instead you'd want to call e.g. `canvas.backend._vispy_mouse_press(...)` directly, which would simulate what happens when `vispy` gets events from the backend. See this for an example:

https://github.com/vispy/vispy/blob/master/vispy/app/backends/_qt.py#L326

Thanks!

For headless rendering stuff, I assume the `before_script` bit in travis is enough and can be reused as is in another project?

Yeah you just need to spin up an `Xvfb` instance to get a functional but slow X server, which I've done for several projects that need it.

Is there a way to not install any backend on travis? I get a `RuntimeError: Could not import any of the backends`

Are you asking if it's possible to render without having any backends installed? Currently the answer is no -- that's what the EGL push is for, but it's not there yet.

If you're asking if we test the no-backend case / support, we do -- there is a `deps=minimal` Travis condition which does not install any backend, and the tests should run fine in this case. Any tests that require a backend should be decorated with `requires_application` to avoid hitting the error you describe.

OK so it looks like I have to install a backend in order to test views on travis.

Yeah. Is this for a different repo, or for a `vispy` branch you're working on that isn't a PR yet...?

No it's another project

Pyglet `tip` is probably the easiest, you can get the line from our `.travis.yml`

How do I close the canvas after, say, 1 second? (with qt) Right now `canvas.show()` freezes travis obviously...

Take a look at our example testing code, it probably does something very close to what you want. What actually gets executed for each example is in `_script`:

https://github.com/vispy/vispy/blob/master/vispy/testing/_runners.py#L166

If you change the `for _ in range(5)` to `for _ in range(60)`, it should run for ~1 sec (assuming draws are very fast).

In other words, something like this should work:

```
with canvas as c:
    t0 = time.time()
    while (time.time() - t0) < 1.:
        c.update()
        c.app.process_events()
        time.sleep(1./60.)
```

Excellent, it worked, thanks!

@rossant, want to write up a section on unit testing for the documentation?

@lcampagn I don't have much time right now, unfortunately, but that's definitely something doable during the sprint


Resolving patch:
diff --git a/vispy/gloo/buffer.py b/vispy/gloo/buffer.py
--- a/vispy/gloo/buffer.py
+++ b/vispy/gloo/buffer.py
@@ -430,8 +430,11 @@ def _prepare_data(self, data, convert=False):
         if not isinstance(data, np.ndarray):
             raise ValueError('Data must be a ndarray (got %s)' % type(data))
         if data.dtype.isbuiltin:
-            if convert is True and data.dtype is not np.float32:
+            if convert is True:
                 data = data.astype(np.float32)
+            if data.dtype in (np.float64, np.int64):
+                raise TypeError('data must be 32-bit not %s'
+                                % data.dtype)
             c = data.shape[-1] if data.ndim > 1 else 1
             if c in [2, 3, 4]:
                 if not data.flags['C_CONTIGUOUS']:
diff --git a/vispy/util/fetching.py b/vispy/util/fetching.py
--- a/vispy/util/fetching.py
+++ b/vispy/util/fetching.py
@@ -270,14 +270,19 @@ def _fetch_file(url, file_name, print_destination=True):
     temp_file_name = file_name + ".part"
     local_file = None
     initial_size = 0
+    # Checking file size and displaying it alongside the download url
+    n_try = 3
+    for ii in range(n_try):
+        try:
+            data = urllib.request.urlopen(url, timeout=5.)
+        except Exception as e:
+            if ii == n_try - 1:
+                raise RuntimeError('Error while fetching file %s.\n'
+                                   'Dataset fetching aborted (%s)' % (url, e))
     try:
-        # Checking file size and displaying it alongside the download url
-        u = urllib.request.urlopen(url, timeout=5.)
-        file_size = int(u.headers['Content-Length'].strip())
+        file_size = int(data.headers['Content-Length'].strip())
         print('Downloading data from %s (%s)' % (url, sizeof_fmt(file_size)))
-        # Downloading data (can be extended to resume if need be)
         local_file = open(temp_file_name, "wb")
-        data = urllib.request.urlopen(url, timeout=5.)
         _chunk_read(data, local_file, initial_size=initial_size)
         # temp file must be closed prior to the move
         if not local_file.closed:


Test patch:
diff --git a/vispy/gloo/tests/test_buffer.py b/vispy/gloo/tests/test_buffer.py
--- a/vispy/gloo/tests/test_buffer.py
+++ b/vispy/gloo/tests/test_buffer.py
@@ -442,6 +442,9 @@ def test_init_allowed_dtype(self):
             names = V.dtype.names
             assert V.dtype[names[0]].base == dtype
             assert V.dtype[names[0]].shape == (3,)
+        for dtype in (np.float64, np.int64):
+            self.assertRaises(TypeError, VertexBuffer,
+                              np.zeros((10, 3), dtype=dtype))
 
         # Tuple/list is also allowed
         V = VertexBuffer([1, 2, 3])
diff --git a/vispy/testing/__init__.py b/vispy/testing/__init__.py
--- a/vispy/testing/__init__.py
+++ b/vispy/testing/__init__.py
@@ -1,6 +1,44 @@
 # -*- coding: utf-8 -*-
 # Copyright (c) 2014, Vispy Development Team.
 # Distributed under the (new) BSD License. See LICENSE.txt for more info.
+"""
+Testing
+=======
+This module provides functions useful for running tests in vispy.
+
+Tests can be run in a few ways:
+
+    * From Python, you can import ``vispy`` and do ``vispy.test()``.
+    * From the source root, you can do ``make test`` which wraps to
+      a call to ``python make test``.
+
+There are various diffrent testing "modes", including:
+
+    * "full": run all tests.
+    * any backend name (e.g., "glfw"): run application/GL tests using a
+      specific backend.
+    * "nobackend": run tests that do not require a backend.
+    * "examples": run repo examples to check for errors and warnings.
+    * "flake": check style errors.
+
+Examples get automatically tested unless they have a special comment toward
+the top ``# vispy: testskip``. Examples that should be tested should be
+formatted so that 1) a ``Canvas`` class is defined, or a ``canvas`` class
+is instantiated; and 2) the ``app.run()`` call is protected by a check
+if ``__name__ == "__main__"``. This makes it so that the event loop is not
+started when running examples in the test suite -- the test suite instead
+manually updates the canvas (using ``app.process_events()``) for under one
+second to ensure that things like timer events are processed.
+
+For examples on how to test various bits of functionality (e.g., application
+functionality, or drawing things with OpenGL), it's best to look at existing
+examples in the test suite.
+
+The code base gets automatically tested by Travis-CI (Linux) and AppVeyor
+(Windows) on Python 2.6, 2.7, 3.4. There are multiple testing modes that
+use e.g. full dependencies, minimal dependencies, etc. See ``.travis.yml``
+to determine what automatic tests are run.
+"""
 
 from ._testing import (SkipTest, requires_application, requires_img_lib,  # noqa
                       has_backend, requires_pyopengl,  # noqa