nose-devs/nose — Issue #356: Object with doctests AND 'normal' tests will ONLY run doctests
Author: jpellerin
Created: 2011-12-14 16:09:00+00:00
State: open

What steps will reproduce the problem?
1. Create a module (which will be examined for tests) that contains
a function such as:

class MyClass(object):
"""
Here's how you use this class:

>>> foo = MyClass()
>>> foo.doSomeStuff()
'doSomeStuff results...'

blah blah...
"""
__test__ = True # So nose will run tests on class

## blah blah... Define the internals of the class...

def test1(self):
# do some test-stuff...
return

2. Run nosetests on this module with the '--with-doctests' option

What is the expected output? What do you see instead?
I would expect to see BOTH the doctest, AND the 'normal' test1 function
run. Instead, ONLY the doctest is run.

What version of the product are you using? On what operating system?
0.11.1-py2.5
Windows XPx64

Please provide any additional information below.

So, I was trying to debug why some tests of mine had suddenly stopped
running, and I came across a rather distrubing discovery: if a class
contains both doctests AND 'normal' tests, it will ONLY run the doctests
(assuming you have --with-doctest enabled, of course).

On digging into the code a bit, it seems the problem is actually even more
more general - if any 'plugin' finds some tests within an object, only the
plugin-found-tests will be run, and none of the 'normal' tests.

In the example above, if nosetests is called with --with-doctest, then
'test1' will be IGNORED, and only the doctest will be run! Or, (as is more
similar to my case), even if you had 20 such test functions, they would all
be ignored, in favor of your single doctest. (Also of note - this will be
the case even if the '# doctest: +SKIP' option is used on the example.)

The code responsible for this behavior can be found in
nose.loader.TestLoader.makeTest, ~lines 513-518 (nose-0.11.1-py2.5)
Essentially, it checks if any plugins find any tests in the current object,
and if so, simply returns those without checking for any 'normal' tests.

Are there plugins that rely on this behavior? If so, could they be
modified? Or a special-case exception made for doctests? Because I think
the default behavior would be to run BOTH types of tests. After all it's
called WITH-doctest, not ONLY-doctest... not to mention that, if you have
SEPARATE objects, one of which has a doctest, and the other has normal
tests, that it will run both types of tests in that case.

I suspect that there are a number of folks who are getting their tests
hidden because of this behavior, and don't even know it...

Google Code Info:
Issue #: 308
Author: elron...@gmail.com
Created On: 2009-11-22T18:30:16.000Z
Closed On: