internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: saulpw/visidata

Issue:
[develop] 2.x gD CommandLog should behave similar to the 1.5.2 CommandLog
**Small description**
The `gD` `commandlog-all` is meant to be a comprehensive CommandLog for the full session. It provides the functionality of what was the standard CommandLog in 1.5.2.

However, it has some logs that are unexpected.

Examples include:
*  `delete-row` when performed on a CommandLog sheet
*  `cmdlog-all`

**Expected result**
I think that the command filters that are configured to occur on regular `cmdlog-sheet`s should also occur on `cmdlog-all`.

I think it is important that 2.x `cmdlog-all` mimic as much of the old behaviour of 1.5.2 `cmdlog` as possible.

**Actual result with screenshot**
![Screenshot from 2020-04-18 22-13-32](https://user-images.githubusercontent.com/7489659/79680204-ee9a9200-81c1-11ea-9aed-9f7063327b51.png)

**Additional context**
v2.-4

This is a sister issue to #426


Discussion:

Resolving patch:
diff --git a/visidata/cmdlog.py b/visidata/cmdlog.py
--- a/visidata/cmdlog.py
+++ b/visidata/cmdlog.py
@@ -64,6 +64,9 @@ def isLoggableCommand(longname):
             return False
     return True
 
+def isLoggableSheet(sheet):
+    return sheet is not vd.cmdlog and not isinstance(sheet, (OptionsSheet, ErrorSheet))
+
 
 @Sheet.api
 def moveToRow(vs, rowstr):
@@ -169,12 +172,13 @@ def afterExecSheet(self, sheet, escaped, err):
 
         # remove user-aborted commands and simple movements
         if not escaped and isLoggableCommand(vd.activeCommand.longname):
-                self.addRow(vd.activeCommand)  # add to global cmdlog
-                sheet.cmdlog_sheet.addRow(vd.activeCommand)  # add to sheet-specific cmdlog
-                if options.cmdlog_histfile:
-                    if not getattr(vd, 'sessionlog', None):
-                        vd.sessionlog = loadInternalSheet(CommandLog, Path(date().strftime(options.cmdlog_histfile)))
-                    append_tsv_row(vd.sessionlog, vd.activeCommand)
+            if isLoggableSheet(sheet):      # don't record actions on global cmdlog or other internal sheets
+                    self.addRow(vd.activeCommand)  # add to global cmdlog
+            sheet.cmdlog_sheet.addRow(vd.activeCommand)  # add to sheet-specific cmdlog
+            if options.cmdlog_histfile:
+                if not getattr(vd, 'sessionlog', None):
+                    vd.sessionlog = loadInternalSheet(CommandLog, Path(date().strftime(options.cmdlog_histfile)))
+                append_tsv_row(vd.sessionlog, vd.activeCommand)
 
         vd.activeCommand = None
 


Test patch:
Repository: saulpw/visidata

Issue:
[wishlist] pasting data into input(): should not react to newlines
I've come against this a few times now where I've accidentally pasted multi line data into the regex search input. This does two things:
1. When a new line character is pasted, it takes this as and enter which competes the command
2. If there are any characters after the new line these are entered as key commands which could potentially mess up the table

Recreate:
Open a sheet
Copy some text to search for. This will include a new line character
Use / to open the input prompt
Paste the text

I often do this when I copy data from a cell using using zY (the cell data maybe partially hidden). This cell value may contain a new line character that I'm unaware of and upon pasting into the input field searches for everything up to the new line, but then entered a bunch of unintended key combinations which are from the following line.


Discussion:
Ah yeah, so basically you would want the input buffer to be cleared on exit from `input()`.  This would still only get the first line, but at least it wouldn't screw things up afterwards.

Alternatively we could make something like a multiline input mode that requires Ctrl+Enter or something to accept the input.  But this would be more complicated and should actually display multiple lines for editing, which inputline doesn't do.
Yea, the first option would be best. I'm not totally convinced myself that multi line input is necessary, _maybe_ convert special characters to their regex counterpart? So if I pasted something with a newline it'd actually paste it as \n?
That might be a good way to go

Resolving patch:
diff --git a/visidata/_input.py b/visidata/_input.py
--- a/visidata/_input.py
+++ b/visidata/_input.py
@@ -232,6 +232,13 @@ def editText(vd, y, x, w, record=True, display=True, **kwargs):
         status('"%s"' % v)
         if record and vd.cmdlog:
             vd.setLastArgs(v)
+
+    # clear keyboard buffer upon exit from input()
+    # input() stops when it reaches an ENTER, and we do not want the expressions
+    # that follow to register as keystrokes
+    # see issue#585
+    curses.flushinp()
+
     return v
 
 


Test patch:
Repository: saulpw/visidata

Issue:
[help keystrokes] Help keystrokes has longnames for about 20 commands
**Small description**
Keystrokes in HelpSheet display commands not keystrokes for about 20 commands.

**Expected result**
Keystroke values

**Actual result with screenshot**

sheet	longname	keystrokes
TableSheet	go-next-selected	next-selected
TableSheet	go-prev-selected	prev-selected
TableSheet	setcol-clipboard	paste-cells
TableSheet	search-prev	prev-search
TableSheet	setcol-input	edit-cells
TableSheet	setcol-fill	fill-nulls
TableSheet	go-next-value	next-value
TableSheet	go-right-page	page-right
BaseSheet	jump-prev	prev-sheet
TableSheet	go-prev-value	prev-value
BaseSheet	open-new	add-sheet
TableSheet	go-next-null	next-null
TableSheet	go-left-page	page-left
TableSheet	go-prev-null	prev-null
TableSheet	open-row	dive-row
DirSheet	open-row	dive-row
BaseSheet	open-statuses	statuses
Canvas	zoomout-mouse	2097152
BaseSheet	show-status	status

**Steps to reproduce with sample data and a .vd**
Visit the help sheet, and look for the long commands above.

**Additional context**
Please include the version of VisiData. Latest from develop



Discussion:
I think this is a bug related to `alias()`.

Nice catch! I am going to see if it is fixable.

Resolving patch:
diff --git a/visidata/help.py b/visidata/help.py
--- a/visidata/help.py
+++ b/visidata/help.py
@@ -36,7 +36,7 @@ def iterload(self):
         self.revbinds = {}  # [longname] -> keystrokes
         itbindings = vd.bindkeys.iterall()
         for (keystrokes, _), longname in itbindings:
-            if keystrokes not in self.revbinds:
+            if (keystrokes not in self.revbinds) and ('-' not in keystrokes or keystrokes[-1] == '-'):
                 self.revbinds[longname] = keystrokes
 
 


Test patch:
Repository: saulpw/visidata

Issue:
[expand-cols undo] undo does not consistently work for expand-cols
**Small description**
Undo doesn't always work with expand-cols, and can leave columns behind.

**Expected result**
Expanded columns are contracted.

**Actual result with screenshot**
https://asciinema.org/a/AoxbEyqboRGYNTfcx5MfsoSm6

**Steps to reproduce with sample data and a .vd**

The issue seems to happen when many columns are created:
```
jo a=$(jo -a {1..35}) b=$(jo -a {1..100}) | vd -f json
```
expand-cols g(
then undo

contract-col seems to work much better.

**Additional context**
Please include the version of VisiData. Latest version of develop.



Discussion:

Resolving patch:
diff --git a/visidata/cmdlog.py b/visidata/cmdlog.py
--- a/visidata/cmdlog.py
+++ b/visidata/cmdlog.py
@@ -140,6 +140,7 @@ def newRow(self, **fields):
     def beforeExecHook(self, sheet, cmd, args, keystrokes):
         if vd.activeCommand:
             self.afterExecSheet(sheet, False, '')
+            vd.activeCommand = None
 
         colname, rowname, sheetname = '', '', None
         if sheet and not (cmd.longname.startswith('open-') and cmd.longname != 'open-row'):
@@ -193,7 +194,6 @@ def afterExecSheet(self, sheet, escaped, err):
                     vd.sessionlog = loadInternalSheet(CommandLog, Path(date().strftime(options.cmdlog_histfile)))
                 append_tsv_row(vd.sessionlog, vd.activeCommand)
 
-        vd.activeCommand = None
 
     def openHook(self, vs, src):
         r = self.newRow(keystrokes='o', input=src, longname='open-file')


Test patch:
Repository: saulpw/visidata

Issue:
null data in input describe sheet
Hi,
I have VisiData v2.-5dev.

Starting from this input file

```
a,b
,3
1,2
,4
5,5
```

when I open it, and then run `shift + i` I have 

|column  |errors  |nulls  |distinct  |mode  |min  |max  |sum  |median  |mean  |stdev  |
|--------|-------:|------:|---------:|------|-----|-----|-----|--------|-----:|------:|
|a       |0       |0      |3         |      |     |     |     |        |      |       |
|b       |0       |0      |4         |\#ERR |     |     |     |        |      |       |

Why do I have no nulls? If I add `--null-value=""` option I have the same output.

Thank you


Discussion:
Am I the only one that has this behavior? 
No @aborruso! This is a bug specifically with that option, but I want to fix it for *all* the options that have this problem, so it is going to take a bit longer to have a fix. =)

Current workaround is to prefix the option with `-g` to set it globally. The issue is that the function that is calculating whether a cell is Null is pulling from the global options, not from the sheet specific ones.

```
vd file.tsv -g --null-value=""
```

The original issue you ran into, where by default VisiData was not parsing those cells as `None` is expected!

Resolving patch:
diff --git a/visidata/main.py b/visidata/main.py
--- a/visidata/main.py
+++ b/visidata/main.py
@@ -205,7 +205,7 @@ def main_vd():
 
         vs = vd.openSource(p, **opts)
         for k, v in current_args.items():  # apply final set of args to sheets specifically on cli, if not set otherwise #573
-            if not vs.options.is_set(k):
+            if not vs.options.is_set(k, vs):
                 vs.options[k] = v
 
         vd.cmdlog.openHook(vs, vs.source)
diff --git a/visidata/settings.py b/visidata/settings.py
--- a/visidata/settings.py
+++ b/visidata/settings.py
@@ -133,7 +133,7 @@ def keys(self, obj=None):
 
     def _get(self, k, obj=None):
         'Return Option object for k in context of obj. Cache result until any set().'
-        opt = self._cache.get((k, obj), None)
+        opt = self._cache.get((k, obj or vd.sheet), None)
         if opt is None:
             opt = self._opts._get(k, obj)
             self._cache[(k, obj or vd.sheet)] = opt


Test patch:
Repository: saulpw/visidata

Issue:
Cursor jumps upwards from final visible row after moving down a row
Commit c9e8f5ed introduced a jarring cursor movement where the cursor hops between the last and second last row when pressing the key to move downwards (moving upwards is fine).

I think it's just an off by one error. Briefly looking myself, the issue is (at least superficially) solved by subtracting 1 from `self.bottomRowIndex` in https://github.com/saulpw/visidata/blob/c9e8f5edc5044293ca4931de3d976ef10d8e2907/visidata/sheets.py#L551

    self.bottomRowIndex = self.cursorRowIndex - 1

However, this is not tested (other than no crashes so far), and I'm not remotely familiar with the codebase, so this may affect something else I'm unaware of.



Discussion:
Hey @uoee, thanks for making an issue. 

I agree that is disconcerting behaviour. I unfortunately cannot reproduce it. =( 

Could you please give more environment information, like your shell and terminal? Are you using tmate? Which VisiData loader are you using? Is this in single-line mode or multi-line mode? A gif would be helpful; if you are on Linux, you can use Peek to make one.
Ah, never mind, I understand what is happening now. Thanks!
Current behaviour

![Peek 2020-12-08 12-37](https://user-images.githubusercontent.com/7489659/101538662-4acd4480-3952-11eb-9ce5-08d1dba520b5.gif)

Previous behaviour

![Peek 2020-12-08 12-38](https://user-images.githubusercontent.com/7489659/101538685-54ef4300-3952-11eb-8e3f-1a75f8525e0c.gif)


Yes, that is annoying.  The solution is trickier than that, unfortunately, as this code also has to deal with scrolling multi-line rows properly (and the bug it fixes is worse, even if multiline usage is less common).  We'll look into it and come up with some more workable solution.

Resolving patch:
diff --git a/visidata/sheets.py b/visidata/sheets.py
--- a/visidata/sheets.py
+++ b/visidata/sheets.py
@@ -319,7 +319,7 @@ def __copy__(self):
 
     @property
     def bottomRowIndex(self):
-        return max(self._rowLayout.keys()) if self._rowLayout else self.topRowIndex+self.nScreenRows
+        return max(self._rowLayout.keys()) if self._rowLayout else self.topRowIndex+self.nScreenRows-1
 
     @bottomRowIndex.setter
     def bottomRowIndex(self, newidx):
@@ -331,7 +331,7 @@ def bottomRowIndex(self, newidx):
             nrows += h
             i += 1
 
-        self._topRowIndex = newidx-i+2
+        self._topRowIndex = newidx-i+2 if nrows == self.nScreenRows-1 else newidx-self.nScreenRows+1
 
     def __deepcopy__(self, memo):
         'same as __copy__'


Test patch:
Repository: saulpw/visidata

Issue:
no loader for url scheme: postgresql
**Small description**

When attempting to start visidata with
```
vd postgresql:///localdb
```
it fails with
```
Error: no loader for url scheme: postgresql
```

**Expected result**

I would have expected it to work the same way 
```
vd postgres:///localdb
```
works, as [both URL schemes are valid](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING).

P.S.: This is somewhere in between a bug and a feature request. As it's super small and about something existing, I called it a bug. I will raise a PR to fix this shortly.


Discussion:

Resolving patch:
diff --git a/visidata/loaders/postgres.py b/visidata/loaders/postgres.py
--- a/visidata/loaders/postgres.py
+++ b/visidata/loaders/postgres.py
@@ -1,6 +1,6 @@
 from visidata import *
 
-__all__ = ['openurl_postgres', 'openurl_rds', 'PgTable', 'PgTablesSheet']
+__all__ = ['openurl_postgres', 'openurl_postgresql', 'openurl_rds', 'PgTable', 'PgTablesSheet']
 
 option('postgres_schema', 'public', 'The desired schema for the Postgres database')
 
@@ -52,6 +52,9 @@ def openurl_postgres(url, filetype=None):
     return PgTablesSheet(dbname+"_tables", sql=SQL(conn))
 
 
+openurl_postgresql=openurl_postgres
+
+
 class SQL:
     def __init__(self, conn):
         self.conn = conn


Test patch:
Repository: saulpw/visidata

Issue:
vds: cannot export sheets containing a date column
**Small description**

It's not possible to save a sheet to a .vds file if said sheet contains a date column.
This results in the error below.

**Expected result**

It should just work(tm).

**Actual result with screenshot**

```stacktrace
Traceback (most recent call last):
  File "/nix/store/srkr2wnwq95ylmgiadh28p3jiaadl5yw-visidata-2.4/lib/python3.8/site-packages/visidata/threads.py", line 215, in _toplevelTryFunc
    t.status = func(*args, **kwargs)
  File "/nix/store/srkr2wnwq95ylmgiadh28p3jiaadl5yw-visidata-2.4/lib/python3.8/site-packages/visidata/loaders/vds.py", line 32, in save_vds
    fp.write(json.dumps(d)+NL)
  File "/nix/store/4s0h5aawbap3xhldxhcijvl26751qrjr-python3-3.8.9/lib/python3.8/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/nix/store/4s0h5aawbap3xhldxhcijvl26751qrjr-python3-3.8.9/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/nix/store/4s0h5aawbap3xhldxhcijvl26751qrjr-python3-3.8.9/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/nix/store/4s0h5aawbap3xhldxhcijvl26751qrjr-python3-3.8.9/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type date is not JSON serializable
```

**Steps to reproduce with sample data and a .vd**

❯ cat testsheet.vd
```log
sheet	col	row	longname	input	keystrokes	comment
visidata_menu			sheets-stack		S	open Sheets Stack: join or jump between the active sheets on the current stack
sheets		キsheets	add-row		a	append a blank row
sheets	name	キ	edit-cell	testsheet	e	edit contents of current cell
sheets		キtestsheet	open-row		^J	open sheet referenced in current row
testsheet	0		rename-col	testcol	^	edit name of current column
testsheet	testcol		type-date		@	set type of current column to date
testsheet			add-row		a	append a blank row
testsheet	testcol	0	edit-cell	2021-06-14	e	edit contents of current cell
testsheet			save-all	test.vds	g^S	save all sheets to given file or directory)
```


**Additional context**

Problem is present on v2.4 and on the develop branch (commit 3350d9fd8c9e64ebf409deae4b31085d12efeb7f)


Discussion:

Resolving patch:
diff --git a/visidata/loaders/vds.py b/visidata/loaders/vds.py
--- a/visidata/loaders/vds.py
+++ b/visidata/loaders/vds.py
@@ -29,7 +29,7 @@ def save_vds(vd, p, *sheets):
             with Progress(gerund='saving'):
                 for row in vs.iterdispvals(*vs.visibleCols, format=False):
                     d = {col.name:val for col, val in row.items()}
-                    fp.write(json.dumps(d)+NL)
+                    fp.write(json.dumps(d, default=str)+NL)
 
 
 class VdsIndexSheet(IndexSheet):


Test patch:
Repository: saulpw/visidata

Issue:
[unfurl-col] unfurling a column with TypedWrapper halts unfurl
**Small description**
When unfurling a column which has a TypedWrapper a sheet loading halt occurs

**Expected result**
The same response as whatever `options.unfurl_empty` would do for a row without a list.

**Actual result with screenshot**
AttributeError: 'TypedWrapper' object has no attribute 'xyz'
![unfurl](https://user-images.githubusercontent.com/11225502/126625610-ee355763-aa7d-427d-a517-abce6b84adfc.gif)

If you get an unexpected error, please include the full stack trace that you get with `Ctrl-E`.

**Steps to reproduce with sample data and a .vd**
`echo '[{"a":1,"d":{"b":[1,2,3]}},{"a":2,"d":{"c":[1,2,3]}},{"a":3,"d":{"b":[1,2,3]}}]' | vd -f json`

```
sheet	col	row	longname	input	keystrokes	comment
			open-file	-	o	
_	d		addcol-expr	curcol['b']	=	create new column from Python expression, with column names as variables
_	curcol_b_		unfurl-col			row-wise expand current column of lists (e.g. [2]) or dicts (e.g. {3}) within that column

```

**Additional context**
v2.6dev



Discussion:

Resolving patch:
diff --git a/visidata/unfurl.py b/visidata/unfurl.py
--- a/visidata/unfurl.py
+++ b/visidata/unfurl.py
@@ -8,12 +8,14 @@
 
 from collections.abc import Iterable, Mapping
 from visidata import vd, Progress, Sheet, Column, ColumnItem, SettableColumn, SubColumnFunc, asyncthread, clean_to_id
+from visidata import stacktrace, TypedExceptionWrapper
 
 
 vd.option('unfurl_empty', False, 'if unfurl includes rows for empty containers', replay=True)
 
 
 class UnfurledSheet(Sheet):
+    # rowdef: [row, key, sub_value]
     @asyncthread
     def reload(self):
         # Copy over base sheet, using SubColumnFunc
@@ -30,7 +32,16 @@ def reload(self):
         self.rows = []
         unfurl_empty = self.options.unfurl_empty
         for row in Progress(self.source.rows):
-            val = self.source_col.getValue(row)
+            try:
+                val = self.source_col.getValue(row)
+            except Exception as e:
+                e.stacktrace = stacktrace()
+                if unfurl_empty:
+                    # TypedExceptionWrapper allows the use of z^E to see the stacktrace
+                    # the exception on its own lacks clarity
+                    self.addRow([row, TypedExceptionWrapper(None, exception=e), TypedExceptionWrapper(None, exception=e)])
+                else:
+                    vd.exceptionCaught(e)
 
             if not isinstance(val, Iterable) or isinstance(val, str):
                 val = [ val ]


Test patch:
Repository: saulpw/visidata

Issue:
[v2.6.1] Warning on startup, if no .visidata/plugins/
```
   File "/usr/local/lib/python3.9/dist-packages/visidata/settings.py", line 364, in loadConfigAndPlugins
     vd.addGlobals(importlib.import_module(modname).__dict__)                
   File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module   
     return _bootstrap._gcd_import(name[level:], package, level)       
   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import      
   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load      
   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked                            
 ModuleNotFoundError: No module named 'plugins'     
 ```


Discussion:

Resolving patch:
diff --git a/visidata/settings.py b/visidata/settings.py
--- a/visidata/settings.py
+++ b/visidata/settings.py
@@ -363,6 +363,11 @@ def loadConfigAndPlugins(vd, args=AttrDict()):
     for modname in (args.imports or vd.options.imports or '').split():
         try:
             vd.addGlobals(importlib.import_module(modname).__dict__)
+        except ModuleNotFoundError as e:
+            # issue #1131
+            if 'plugins' in e.args[0]:
+                continue
+            vd.exceptionCaught(e)
         except Exception as e:
             vd.exceptionCaught(e)
             continue


Test patch: