Make lexer fail gracefully when memory can't be allocated.

This should address the issue raised in #895 

With this patch, you now get:
```python
>>> import gc
>>> gc.disable()
>>> gc.enable()
MemoryError
```
rather than a fatal error.

We're still hooped, in that no further code can be executed, but you can still hit Control-D and get a soft-reset.