Repository: vispy/vispy

Issue:
FIX: Reduce instances of errors in callbacks
Maybe we should have a registry like `warnings` does that prevents error massages from being printed again. It makes debugging very difficult to have the same error message printed every `on_draw` call... thoughts?

Relevent for @nmz787's comment in #213.



Discussion:
@nmz787, responding to your post in #213 over here since this is the more relevant place to put it.

I know the type of behavior you mean, where you get an error in the `on_initialize` or `on_draw`, and subsequent calls to `on_draw` are broken and thus spew a ton of errors.

I'm not sure if it's possible to throw an error at the application level and terminate in the case of a failed `on_initialize` or `on_draw` callback, due to how event loops are handled at the application level. @almarklein do you know?

It should definitely be possible, however, to have a type of registry system (like the `warnings` package uses) where a given error message is only reported once, even if it doesn't prevent continued execution. I'm also not sure that we'd want to discontinue execution in _all_ cases when a callback fails, so we probably want to implement a registry-like behavior in any case.

Maybe a dictionary where the exception string is the key, and the value is a counter. A message only prints if it isn't in the dictionary, otherwise it just increments the value stored. From what I've read it's fastest (computational time) to implement something with a try except block:

```
def printWarning(messageString):
    try:
        warnings[messageString]+=1
    else:
        warnings[messageString]=1
        print messageString
```

then maybe write those data to a log file on exit?

```
def ExitHandler():
    with open('vispy_warning_error.log','a+') as warningFile:
        for message in warnings:
            warningFile.write('The following message was emitted %s times.\n' % warnings[message])
            warningFile.write(str(message))
            warningFile.write('\n----\n')
atexit.register(ExitHandler)
```

That seems reasonable. We can probably get away with not logging for a first pass. (We'd want it to be optional, we'd also want to think about logging everything from `vispy.util.logger`, etc. -- it's a bit of a can of worms there.)

Is a `try` actually faster than doing `if messageString not in warnings:`??? That would be weird, but good to know. I think something like that would work well. Let's wait to hear from the other devs before implementing anything, though. IIRC there is some control over the callback error messages already, but I don't remember what it is. I can look it up if the other devs don't chime in about it.

@lcampagn you've worked with the events system a bit -- any thoughts?

`warnings` looks like it could be good, but I've never worked with it before. I guess I was wrong about `try/except` it being 'fastest', I believe I was influenced by this post a few months ago:
Python dictionary: add or increment entry
http://stackoverflow.com/a/2627871/253127

Just to be clear, I don't actually want to use the `warnings` package, but rather implement something like it does where we use our own callback error registry. We shouldn't shadow the name `warnings` either (since it's a built-in package name), so we should call the registry dict something like `_vispy_callback_errors` or so if we do decide to go this route.

> I'm not sure if it's possible to throw an error at the application level and terminate in the case of a failed on_initialize or on_draw callback,

You _could_ do sys.exit() but I would not recommend that, since that means we cannot go into postmortem debugging.

> Is a try actually faster than doing if messageString not in warnings:??? 

`try` is faster than `if` when it does not trigger an exception. If it does, it is much slower.

Well the exception happens once, the try is successful probably 1000s of times in my last case.

Not too keen on trying to hide repeated error messages. This can lead to a lot of frustration in debugging if you're not aware this is happening. Adding another message like "Suppressed N repeated error messages" might alleviate this concern..

Another approach is to implement error checking in the individual examples that would disable further calls to the draw event. 

I'm fine with adding a subsequent message about suppression. Even if that
one-line suppression message were printed instead of the full traceback it
would be a great improvement. Then I'd only get 60 error lines per second
printed instead of about 600 (behavior right now), which makes debugging a
huge pain.

Another approach: For some events (like paint) we could disconnect the event handler when it raises an error (and notify that we do this, of course)

Yes--`EventEmitter` currently has two properties:
- `print_callback_errors` causes the traceback to be logged for any callbacks that raise an exception
- `ignore_callback_errors` causes the emitter to consume callback exceptions, allowing it to continue processing other callbacks. 

These might be replaced by a single method like 

```
def set_error_behavior(ignore=True, log=True, disconnect=False):
```

Great idea.

I don't like our current behavior as default because it makes debugging
very difficult. Should we make disconnect True by default? If not, then we
should probably still do the registry to tame the output a bit. In fact, we
might want to have support for a registry anyway. WDYT?

> Should we make disconnect True by default?

Probably not.. one of the nice things about event-driven programming is its tolerance for intermittent failures. Making this available as a config / command-line option is a good idea, though. In fact, a global `debug` option that sets multiple appropriate flags might be nice..

> If not, then we should probably still do the registry to tame the output a bit.

Yet another option is to allow something like

```
emitter.set_error_handling(log='once')
```

I am ok with adding a registry if other options are not sufficient. If you do implement this:
- Definitely need a config option and command line flag for disabling it.
- Can we somehow ensure that "suppressed error message" messages are not buried in non-suppressed error messages? 
- What are the criteria for deciding whether an error message has already been seen? 

A config / command-line flag should be doable. IIRC we already set the logging level on startup, in which case this would be hopefully a straightforward extension of that.

In terms of the short suppression lines getting buried, I am not too worried about this case. If there are really _that_ many error messages in a user's code, then I wouldn't consider having the `"Suppressed repeated message for the Nth time" be uber-visible a high priority.

To start, I'd just use strict string equivalence of the error message for the registry. We can add a tolerance / make it better later.

> To start, I'd just use strict string equivalence of the error message for the registry. We can add a tolerance / make it better later.

Sounds expensive--many of these tracebacks are quite large. There's also the possibility that the registry could consume memory quickly. Fuzzy matching would help avoid this, at even greater expense.. 

Perhaps we should just try it for now. So long as the default behavior is not affected (especially with respect to performance), I see no problem.

Let's see how it works. The performance hits should only happen when there are errors, which isn't too bad a place to have a performance hit. The current behavior (printing tons of lines) has actually bogged down my IDE before due to the printing, which is itself a form of performance hit :)

I bepeive the logger module can be handed a file or a stream, which sys.stderr is.


Resolving patch:
diff --git a/vispy/util/_logging.py b/vispy/util/_logging.py
--- a/vispy/util/_logging.py
+++ b/vispy/util/_logging.py
@@ -7,6 +7,7 @@
 import inspect
 import re
 import traceback
+from functools import partial
 
 from ..ext.six import string_types
 
@@ -63,8 +64,7 @@ class _VispyStreamHandler(logging.StreamHandler):
     Prepending of traceback information is done in _VispyFormatter.
     """
     def __init__(self):
-        #logging.StreamHandler.__init__(self, _WrapStdOut())
-        logging.StreamHandler.__init__(self, sys.stdout)
+        logging.StreamHandler.__init__(self, sys.stderr)
         self._vispy_formatter = _lf
         self.setFormatter(self._vispy_formatter)
         self._vispy_match = None
@@ -76,17 +76,13 @@ def _vispy_emit_match_andor_record(self, record):
         """Log message emitter that optionally matches and/or records"""
         test = record.getMessage()
         match = self._vispy_match
-        if (match is None or re.search(match, test) or 
+        if (match is None or re.search(match, test) or
                 re.search(match, _get_vispy_caller())):
             if self._vispy_emit_record:
                 fmt_rec = self._vispy_formatter.format(record)
                 self._vispy_emit_list.append(fmt_rec)
             return logging.StreamHandler.emit(self, record)
 
-    def _vispy_emit(self, record):
-        """Log message emitter that wraps directly to the standard method"""
-        return logging.StreamHandler.emit(self, record)
-
     def _vispy_set_match(self, match):
         old_match = self._vispy_match
         self._vispy_match = match
@@ -94,7 +90,7 @@ def _vispy_set_match(self, match):
         if match is not None or self._vispy_emit_record:
             self.emit = self._vispy_emit_match_andor_record
         else:
-            self.emit = self._vispy_emit
+            self.emit = partial(logging.StreamHandler.emit, self)
         return old_match
 
     def _vispy_set_emit_record(self, record):
@@ -104,7 +100,7 @@ def _vispy_set_emit_record(self, record):
         if match is not None or self._vispy_emit_record:
             self.emit = self._vispy_emit_match_andor_record
         else:
-            self.emit = self._vispy_emit
+            self.emit = partial(logging.StreamHandler.emit, self)
 
     def _vispy_reset_list(self):
         self._vispy_emit_list = list()
diff --git a/vispy/util/event.py b/vispy/util/event.py
--- a/vispy/util/event.py
+++ b/vispy/util/event.py
@@ -18,6 +18,7 @@
 import inspect
 import weakref
 import traceback
+import math
 
 from .ordereddict import OrderedDict
 from ._logging import logger
@@ -190,6 +191,7 @@ def __init__(self, source=None, type=None, event_class=Event):
         self._emitting = False  # used to detect emitter loops
         self.source = source
         self.default_args = {}
+        self._err_registry = {}
         if type is not None:
             self.default_args['type'] = type
 
@@ -197,7 +199,7 @@ def __init__(self, source=None, type=None, event_class=Event):
         self.event_class = event_class
 
         self._ignore_callback_errors = True
-        self._print_callback_errors = True
+        self.print_callback_errors = 'reminders'
 
     @property
     def ignore_callback_errors(self):
@@ -216,6 +218,10 @@ def ignore_callback_errors(self, val):
     def print_callback_errors(self):
         """Print a message and stack trace if a callback raises an exception
 
+        Valid values are "first" (only show first instance), "reminders" (show
+        complete first instance, then counts), "always" (always show full
+        traceback), or "never".
+
         This assumes ignore_callback_errors=True. These will be raised as
         warnings, so ensure that the vispy logging level is set to at
         least "warning".
@@ -224,6 +230,9 @@ def print_callback_errors(self):
 
     @print_callback_errors.setter
     def print_callback_errors(self, val):
+        if val not in ('first', 'reminders', 'always', 'never'):
+            raise ValueError('print_callback_errors must be "first", '
+                             '"reminders", "always", or "never"')
         self._print_callback_errors = val
 
     @property
@@ -432,10 +441,32 @@ def _invoke_callback(self, cb, event):
             del tb  # Get rid of it in this namespace
             # Handle
             if self.ignore_callback_errors:
-                if self.print_callback_errors:
-                    logger.log_exception()
-                    logger.warning("Error invoking callback %s for "
-                                   "event: %s" % (cb, event))
+                if self.print_callback_errors != "never":
+                    this_print = 'full'
+                    if self.print_callback_errors in ('first', 'reminders'):
+                        # need to check to see if we've hit this yet
+                        key = repr(cb) + repr(event)
+                        if key in self._err_registry:
+                            self._err_registry[key] += 1
+                            if self.print_callback_errors == 'first':
+                                this_print = None
+                            else:  # reminders
+                                ii = self._err_registry[key]
+                                # Use logarithmic selection
+                                # (1, 2, ..., 10, 20, ..., 100, 200, ...)
+                                if ii % (10 ** int(math.log10(ii))) == 0:
+                                    this_print = ii
+                                else:
+                                    this_print = None
+                        else:
+                            self._err_registry[key] = 1
+                    if this_print == 'full':
+                        logger.log_exception()
+                        logger.warning("Error invoking callback %s for "
+                                       "event: %s" % (cb, event))
+                    elif this_print is not None:
+                        logger.warning("Error invoking callback %s repeat %s"
+                                       % (cb, this_print))
             else:
                 raise
 


Test patch:
diff --git a/vispy/util/tests/test_event_emitter.py b/vispy/util/tests/test_event_emitter.py
--- a/vispy/util/tests/test_event_emitter.py
+++ b/vispy/util/tests/test_event_emitter.py
@@ -245,7 +245,7 @@ def test_chained_emitters(self):
     def test_emitter_error_handling(self):
         """Emitter error handling"""
         em = EventEmitter(type='test_event')
-        em.print_callback_errors = False
+        em.print_callback_errors = 'never'
 
         def cb(ev):
             raise Exception('test')