Repository: xonsh/xonsh

Issue:
AttributeError exception in JSON history backend
## xonfig

<details>

```
$ xonfig
+------------------+---------------------+
| xonsh            | 0.9.18              |
| Git SHA          | 292d292d            |
| Commit Date      | May 1 13:35:50 2020 |
| Python           | 3.8.2               |
| PLY              | 3.11                |
| have readline    | True                |
| prompt toolkit   | 3.0.5               |
| shell type       | prompt_toolkit      |
| pygments         | 2.6.1               |
| on posix         | True                |
| on linux         | True                |
| distro           | arch                |
| on darwin        | False               |
| on windows       | False               |
| on cygwin        | False               |
| on msys2         | False               |
| is superuser     | False               |
| default encoding | utf-8               |
| xonsh encoding   | utf-8               |
| encoding errors  | surrogateescape     |
+------------------+---------------------+
```

</details>

## Current Behavior
If `$HISTCONTROL` is set to either `ignoreerr`, `ignoredups` or both, JSON history backend may crash with the backtrace below. Though the backtrace was triggered by Powerline xontrib, it happens whenever something attempts to reach the end of the history (e.g. `__xonsh__.history.rtns[-1]`) of an already flushed history buffer. This is due to the fact that not all history entries are always flushed to the history file, but the change of the history length, caused by skipping some entries, is not reflected by the internal state (it simply thinks that the history is longer than it really is).

### Traceback (if applicable)

<details>

```
Traceback (most recent call last):
  File "/home/str/code/xonsh/xonsh/prompt/base.py", line 47, in __call__
    prompt = self._format_prompt(template=template)
  File "/home/str/code/xonsh/xonsh/prompt/base.py", line 55, in _format_prompt
    template = template() if callable(template) else template
  File "/usr/lib/python3.8/site-packages/xontrib/powerline.xsh", line 170, in prompt
    r = s()
  File "/usr/lib/python3.8/site-packages/xontrib/powerline.xsh", line 105, in rtns
    if __xonsh__.history.rtns and __xonsh__.history.rtns[-1] != 0:
  File "/home/str/code/xonsh/xonsh/history/json.py", line 273, in __getitem__
    rtn = lj["cmds"][key].get(self.field, self.default)
AttributeError: 'list' object has no attribute 'get'
```

</details>

## Steps to Reproduce

```
$ __xonsh__.history.buffersize = 1
$ $XONSH_SHOW_TRACEBACK=True
$ $XONSH_DEBUG=True
$ foobarfoobar # returns error
$ __xonsh__.history.rtns[-1]
Traceback (most recent call last):
  File "/home/str/code/xonsh/xonsh/base_shell.py", line 362, in default
    run_compiled_code(code, self.ctx, None, "single")
  File "/home/str/code/xonsh/xonsh/codecache.py", line 67, in run_compiled_code
    func(code, glb, loc)
  File "<xonsh-code>", line 1, in <module>
  File "/home/str/code/xonsh/xonsh/history/json.py", line 274, in __getitem__
    rtn = lj["cmds"][key].get(self.field, self.default)
AttributeError: 'list' object has no attribute 'get'
```


Discussion:
There's another issue with the JSON history backend: History buffer flushing happens asynchronously in a background thread, but the buffer gets cleared immediately after it is handed over to the flusher thread constructor, so the history gets most probably truncated.

Resolving patch:
diff --git a/xonsh/history/json.py b/xonsh/history/json.py
--- a/xonsh/history/json.py
+++ b/xonsh/history/json.py
@@ -222,7 +222,9 @@ def files(self, only_unlocked=False):
 class JsonHistoryFlusher(threading.Thread):
     """Flush shell history to disk periodically."""
 
-    def __init__(self, filename, buffer, queue, cond, at_exit=False, *args, **kwargs):
+    def __init__(
+        self, filename, buffer, queue, cond, at_exit=False, skip=None, *args, **kwargs
+    ):
         """Thread for flushing history."""
         super(JsonHistoryFlusher, self).__init__(*args, **kwargs)
         self.filename = filename
@@ -231,6 +233,7 @@ def __init__(self, filename, buffer, queue, cond, at_exit=False, *args, **kwargs
         queue.append(self)
         self.cond = cond
         self.at_exit = at_exit
+        self.skip = skip
         if at_exit:
             self.dump()
             queue.popleft()
@@ -255,9 +258,13 @@ def dump(self):
         for cmd in self.buffer:
             if "ignoredups" in opts and cmd["inp"] == last_inp:
                 # Skipping dup cmd
+                if self.skip is not None:
+                    self.skip(1)
                 continue
             if "ignoreerr" in opts and cmd["rtn"] != 0:
                 # Skipping failed cmd
+                if self.skip is not None:
+                    self.skip(1)
                 continue
             cmds.append(cmd)
             last_inp = cmd["inp"]
@@ -371,6 +378,7 @@ def __init__(self, filename=None, sessionid=None, buffersize=100, gc=True, **met
         self._queue = collections.deque()
         self._cond = threading.Condition()
         self._len = 0
+        self._skipped = 0
         self.last_cmd_out = None
         self.last_cmd_rtn = None
         meta["cmds"] = []
@@ -385,7 +393,7 @@ def __init__(self, filename=None, sessionid=None, buffersize=100, gc=True, **met
         self.rtns = JsonCommandField("rtn", self)
 
     def __len__(self):
-        return self._len
+        return self._len - self._skipped
 
     def append(self, cmd):
         """Appends command to history. Will periodically flush the history to file.
@@ -427,10 +435,19 @@ def flush(self, at_exit=False):
         """
         if len(self.buffer) == 0:
             return
+
+        def skip(num):
+            self._skipped += num
+
         hf = JsonHistoryFlusher(
-            self.filename, tuple(self.buffer), self._queue, self._cond, at_exit=at_exit
+            self.filename,
+            tuple(self.buffer),
+            self._queue,
+            self._cond,
+            at_exit=at_exit,
+            skip=skip,
         )
-        self.buffer.clear()
+        self.buffer = []
         return hf
 
     def items(self, newest_first=False):


Test patch: