Repository: ipython/ipython

Issue:
TypeError: must be string without null bytes, not str
Hi, I  just installed iPython 2.7 under macports, and when I try to start it I get the following error:

$ /opt/local/bin/ipython-2.7 Traceback (most recent call last):
  File "/opt/local/bin/ipython-2.7", line 7, in <module>
    launch_new_instance()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/frontend/terminal/ipapp.py", line 388, in launch_new_instance
    app.initialize()
  File "<string>", line 2, in initialize
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/config/application.py", line 84, in catch_config_error
    return method(app, _args, *_kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/frontend/terminal/ipapp.py", line 324, in initialize
    self.init_shell()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/frontend/terminal/ipapp.py", line 340, in init_shell
    ipython_dir=self.ipython_dir)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/config/configurable.py", line 318, in instance
    inst = cls(_args, *_kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/frontend/terminal/interactiveshell.py", line 360, in **init**
    user_module=user_module, custom_exceptions=custom_exceptions
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 481, in **init**
    self.init_readline()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 1868, in init_readline
    self.refill_readline_hist()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 1886, in refill_readline_hist
    stdin_encoding))
TypeError: must be string without null bytes, not str

If you suspect this is an IPython bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@scipy.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    c.Application.verbose_crash=True

Can you help ? I am not sure what is wrong here.



Discussion:
If you move your `history.sqlite` out of the way (don't delete it, we might want to check it to work out the problem), does it start OK? Run `ipython locate profile` to see where that file is located.

I hit this too. Merely typing 'ipython' would yield the error in the original question. ipython 0.13.1, python 2.7.3.

Moving history.sqlite (in ~/.config/ipython/profile_default/) out of the way solved the problem.

This situation arose immediately after piping data into a script that did IPython.embed(). Might this have poisoned the history database. I see a bunch of illegible characters (not displayable in utf-8) in the history table.

Can you send us a copy of the corrupt history.sqlite file? It has every command you've used in IPython 0.11+, so if it might contain confidential data that you don't want to send, we'll work out another way to investigate it.

It's possible that piping data in confused it. Can you produce a small example that reproduces the problem?

I stripped the history to a minimal case that demonstrates the problem. See http://harts.net/reece/tmp/ipython-corrupt-history/ for the history file and a session log. Thanks for investigating.


Resolving patch:
diff --git a/IPython/core/interactiveshell.py b/IPython/core/interactiveshell.py
--- a/IPython/core/interactiveshell.py
+++ b/IPython/core/interactiveshell.py
@@ -1862,14 +1862,20 @@ def refill_readline_hist(self):
             # Ignore blank lines and consecutive duplicates
             cell = cell.rstrip()
             if cell and (cell != last_cell):
-                if self.multiline_history:
-                      self.readline.add_history(py3compat.unicode_to_str(cell,
-                                                                stdin_encoding))
-                else:
-                    for line in cell.splitlines():
-                        self.readline.add_history(py3compat.unicode_to_str(line,
-                                                                stdin_encoding))
-                last_cell = cell
+                try:
+                    if self.multiline_history:
+                          self.readline.add_history(py3compat.unicode_to_str(cell,
+                                                                    stdin_encoding))
+                    else:
+                        for line in cell.splitlines():
+                            self.readline.add_history(py3compat.unicode_to_str(line,
+                                                                    stdin_encoding))
+                    last_cell = cell
+                
+                except TypeError:
+                    # The history DB can get corrupted so it returns strings
+                    # containing null bytes, which readline objects to.
+                    continue
 
     @skip_doctest
     def set_next_input(self, s):


Test patch: