Repository: xonsh/xonsh

Issue:
Up arrow history search stops accepting new history if __xonsh_history__.buffersize = 0
I changed it yesterday morning and then my "Up arrow history" froze there. When I removed the `__xonsh_history__.buffersize = 0` line from my .xonshrc my "Up arrow history" started saving commands again.

As I've been working on my getHistory() command I've been able to monitor the history in XONSH_DATA_DIR and everything is fine there, it's saving everything exactly as I would expect it. 

So these must be two different mechanisms and when __xonsh_history_.flush() occurs after every command it's not additionally appending the command to the "Up arrow history", which I can only assume is only taking place when xonsh exits successfully.



Discussion:
Hi @JohnLunzer - the "Up arrow history" this is a feature of the readline and prompt-toolkit shells. They each have their own mechanisms for loading history into their history buffers, see:
- https://github.com/scopatz/xonsh/blob/master/xonsh/readline_shell.py#L382
- https://github.com/scopatz/xonsh/blob/master/xonsh/ptk/history.py

The history is read in separately from how it is saved.  The potential trip up is that the `history` command will also go read in the history file of the current session if you ask for data that has already been flushed.

This `files = hist.gc.unlocked_files()` is common code between them. Here is the bug, I don't know where but when `__xonsh_history__.buffersize = 0` the history files are not being unlocked when xonsh exits successfully. 

I have been facing a maybe related problem. Sometimes the history (ptk) is not displayed when I press up arrow. Typing one character and deleting it re-enable the feature for me

Hi @JohnLunzer - The reason for the lock is so that xonsh history garbage collection (gc) from one xonsh process doesn't remove the history from another xonsh process that is also currently running. It is not meant, per se to restrict reading history from such files, though I believe that the way things are presently implemented it does that as well.

If the lock status was not aimed at history then I am of the opinion that it should not be checked for any history commands, right now we're throwing away valid history. I believe this can be fixed rather easily by renaming (all references of) `hist.gc.unlocked_files()` to `hist.gc.files()` with a new parameter `locked=True` as default. `hist.gc.files(locked=False)` would mimic the original functionality (which we wouldn't be using). This should be a pretty straightforward change without unwanted side effects. 


Resolving patch:
diff --git a/xonsh/history.py b/xonsh/history.py
--- a/xonsh/history.py
+++ b/xonsh/history.py
@@ -90,7 +90,7 @@ def run(self):
             hsize, units = env.get('XONSH_HISTORY_SIZE')
         else:
             hsize, units = to_history_tuple(self.size)
-        files = self.unlocked_files()
+        files = self.files(only_unlocked=True)
         rmfiles_fn = self.gc_units_to_rmfiles.get(units)
         if rmfiles_fn is None:
             raise ValueError('Units type {0!r} not understood'.format(units))
@@ -101,8 +101,9 @@ def run(self):
             except OSError:
                 pass
 
-    def unlocked_files(self):
-        """Find and return the history files that are unlocked.
+    def files(self, only_unlocked=False):
+        """Find and return the history files. Optionally locked files may be
+        excluded.
 
         This is sorted by the last closed time. Returns a list of (timestamp,
         file) tuples.
@@ -115,10 +116,12 @@ def unlocked_files(self):
         for f in fs:
             try:
                 lj = lazyjson.LazyJSON(f, reopen=False)
-                if lj['locked']:
+                if only_unlocked and lj['locked']:
                     continue
                 # info: closing timestamp, number of commands, filename
-                files.append((lj['ts'][1], len(lj.sizes['cmds']) - 1, f))
+                files.append((lj['ts'][1] or time.time(), 
+                              len(lj.sizes['cmds']) - 1, 
+                              f))
                 lj.close()
             except (IOError, OSError, ValueError):
                 continue
diff --git a/xonsh/ptk/history.py b/xonsh/ptk/history.py
--- a/xonsh/ptk/history.py
+++ b/xonsh/ptk/history.py
@@ -53,7 +53,7 @@ def run(self):
         ptkhist = self.ptkhist
         while self.wait_for_gc and hist.gc.is_alive():
             time.sleep(0.011)  # gc sleeps for 0.01 secs, sleep a beat longer
-        files = hist.gc.unlocked_files()
+        files = hist.gc.files()
         for _, _, f in files:
             try:
                 lj = lazyjson.LazyJSON(f, reopen=False)
diff --git a/xonsh/readline_shell.py b/xonsh/readline_shell.py
--- a/xonsh/readline_shell.py
+++ b/xonsh/readline_shell.py
@@ -406,7 +406,7 @@ def run(self):
         hist = builtins.__xonsh_history__
         while self.wait_for_gc and hist.gc.is_alive():
             time.sleep(0.011)  # gc sleeps for 0.01 secs, sleep a beat longer
-        files = hist.gc.unlocked_files()
+        files = hist.gc.files()
         i = 1
         for _, _, f in files:
             try:


Test patch: