internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: saulpw/visidata

Issue:
[zo] Give feedback when a file or URL is not found in the cell value
New users (and me when I have a few z related `open-cell-XYZ` commands 🙃) may get confused at the purpose of `zo`, when it is usually `zENTER` they want.

I propose the change, whereby a status is given to the user to tell them that a `file` or `URL` wasn't found (and therefore nothing happens). Also maybe suggest they want `zENTER`?


Discussion:

Resolving patch:
diff --git a/visidata/_open.py b/visidata/_open.py
--- a/visidata/_open.py
+++ b/visidata/_open.py
@@ -132,4 +132,4 @@ def loadInternalSheet(vd, cls, p, **kwargs):
 
 
 BaseSheet.addCommand('o', 'open-file', 'vd.push(openSource(inputFilename("open: "), create=True))', 'Open file or URL')
-TableSheet.addCommand('zo', 'open-cell-file', 'vd.push(openSource(cursorDisplay))', 'Open file or URL from path in current cell')
+TableSheet.addCommand('zo', 'open-cell-file', 'vd.push(openSource(cursorDisplay) or fail(f"file {cursorDisplay} does not exist"))', 'Open file or URL from path in current cell')


Test patch:
Repository: saulpw/visidata

Issue:
First row in the All Sheets Sheet is not sorted.
**Small description**
First row is not  sorted with [  or  ]  in columns of Sheets Sheet.

**Expected result**
All rows should be sorted in Sheets Sheet.

**Actual result with screenshot( sort in nRows columns)**
![222127624](https://user-images.githubusercontent.com/87310483/206032798-d8fbb524-6556-4493-b2a5-44826ef840b5.png)
![22212755939](https://user-images.githubusercontent.com/87310483/206032868-f61822d8-763f-4c2b-ad0f-b41d47e9bde2.png)

**Steps to reproduce with sample data and a .vd**
1. open several files.
2. press shift + S to open Sheets Sheet.
3. go to nRows or nCols or other numeric columns, and type [  or  ] to sort.

**Additional context**
Version 2.10.2



Discussion:
This is actually by design on the Sheets Sheet, since the first sheet is the current/top sheet, so sorting the list of sheets would surprisingly change the current sheet. The issue that you reported on reddit, is that the global Sheets Sheet has the same behavior, even though its first sheet is not the top sheet.
Sure , I have to moodify my report as follows:

**Small description**
First row is not  sorted with [  or  ]  in columns of  ***global*** Sheets Sheet.

**Expected result**
All rows should be sorted in ***global*** Sheets Sheet.

**Actual result with screenshot( sort in nRows columns)**
![2221277857](https://user-images.githubusercontent.com/87310483/206044802-bc59c92c-e55c-4c5f-add2-22daeab127ba.png)
![2221277839](https://user-images.githubusercontent.com/87310483/206044804-65cc9738-3bd2-404a-893f-a26108f1b2ad.png)

**Steps to reproduce with sample data and a .vd**
1. open several files.
2. press ` g  shift + S` to open ***global*** Sheets Sheet.
3. go to nRows or nCols or other numeric columns, and type [  or  ] to sort.

**Additional context**
Version 2.10.2


Resolving patch:
diff --git a/visidata/sheets.py b/visidata/sheets.py
--- a/visidata/sheets.py
+++ b/visidata/sheets.py
@@ -1011,6 +1011,10 @@ def reload(self):
     def sort(self):
         self.rows[1:] = sorted(self.rows[1:], key=self.sortkey)
 
+class GlobalSheetsSheet(SheetsSheet):  #1620
+    def sort(self):
+        IndexSheet.sort(self)
+
 @VisiData.property
 @drawcache
 def _evalcontexts(vd):
@@ -1067,7 +1071,7 @@ def push(vd, vs, pane=0, load=True):
 
 @VisiData.lazy_property
 def allSheetsSheet(vd):
-    return SheetsSheet("sheets_all", source=vd.allSheets)
+    return GlobalSheetsSheet("sheets_all", source=vd.allSheets)
 
 @VisiData.lazy_property
 def sheetsSheet(vd):


Test patch:
Repository: saulpw/visidata

Issue:
Create New Table from Clipboard
It would be nice to have a command which can create a new table directly from the clipboard.

For example, suppose I have tab separated data in my clipboard.
I would like to be able to execute a visidata command to load a table from the clipboard.
The command might ask me what filetype I am loading (tsv in this case).
Then it would load the data directly from the clipboard.

This would be particularly handy for quick file format conversions. e.g. if I copy a table from OneNote and want to convert it to a Markdown table, I could just import it into Visidata form the clipboard, and copy it back to the clipboard as a markdown table.


Discussion:
This sounds like a great idea!
I do something similar at the moment on Mac. I use the following to take in clipboard data, the same might exist for Windows/Powershell?

`pbpaste | vd -f json -`
Windows has a copy command (`clip.exe`), but not a paste command.
However, you should be able to do:

```
echo 'paste content here' | vd -f tsv -
```

(I forgot the trailing `-` when I tried it earlier.)
I just pushed an `open-syspaste` command which I think is a good start on what you want. 

@daviewales Would you be up for sponsoring this feature?  It could use a sentence or two docstring at the top of the file, and if you wanted it to ask for an input filetype, you could extend it to do that too.  And any other little tweaks that might be useful.

(I put it in the new `features/` toplevel folder to indicate that it's a standalone little feature.  We're importing them all by default for now, and may provide an option to enable/disable them individually in the future.)
Thanks @saulpw! I'll have a look. 😄 I would definitely use the input filetype, so I'll see if I can figure it out.

Resolving patch:
diff --git a/visidata/features/open_syspaste.py b/visidata/features/open_syspaste.py
--- a/visidata/features/open_syspaste.py
+++ b/visidata/features/open_syspaste.py
@@ -1,3 +1,7 @@
+'''
+Load new table from system clipboard
+'''
+
 from visidata import vd, BaseSheet, Path
 
 
@@ -11,4 +15,4 @@ def open_syspaste(sheet, filetype='tsv'):
     return vd.openSource(p, filetype=filetype)
 
 
-BaseSheet.addCommand('', 'open-syspaste', 'vd.push(open_syspaste())', 'open clipboard as tsv')
+BaseSheet.addCommand('gShift+P', 'open-syspaste', 'vd.push(open_syspaste(filetype=vd.input("paste as filetype: ", value="tsv")))', 'open clipboard as filetype')


Test patch:
Repository: saulpw/visidata

Issue:
Bugs with save-all
        I've a question about saving.

1. I load a two sheet notebook "responses-grades-cda.xlsx".
2. `g ctrl+s`
3. it gives me a blank line -- why not default to name of file without xlsx extension?
4. I type in file name without extension + `vds`: "responses-grades-cda.vds".
5. I get an error: "AttributeError: 'X1sxIndexSheet' object has no attribute 'join'"

_Originally posted by @reagle in https://github.com/saulpw/visidata/discussions/1266#discussioncomment-4870711_
      


Discussion:
Hi @reagle,

`g Ctrl+S` requires that the sheets you want to save are selected (`gs` will select them all). This is why the input is blank, the full status is `saving 0 sheets to: _____`. 

The stacktrace is also related to no sheets being selected. I am going to add a warning, and try to understand the stacktrace.

Resolving patch:
diff --git a/visidata/save.py b/visidata/save.py
--- a/visidata/save.py
+++ b/visidata/save.py
@@ -104,6 +104,10 @@ def save_cols(vd, cols):
 def saveSheets(vd, givenpath, *vsheets, confirm_overwrite=False):
     'Save all *vsheets* to *givenpath*.'
 
+    if not vsheets: # blank tuple
+        vd.warning('no sheets to save')
+        return
+
     filetype = givenpath.ext or options.save_filetype
 
     vd.clearCaches()


Test patch:
Repository: saulpw/visidata

Issue:
Plot uses wrong range for y-axis
**Small description**

I open a plot for a column, and VisiData sets the maximum y for 1.17:

![screenshot_TURING_2023-01-24_204539](https://user-images.githubusercontent.com/56778/214381535-e57d54f9-e72a-4a32-b75e-e06eb31268c4.jpg)


Even though the maximum value for that column is 0.16:

![screenshot_TURING_2023-01-24_204611](https://user-images.githubusercontent.com/56778/214381574-a98b25b9-bbc5-41ff-be23-593415d4ce3e.jpg)

JSONL file: https://www.dropbox.com/s/n082mwbv7j85cee/progress.jsonl?dl=1


Discussion:

Resolving patch:
diff --git a/visidata/canvas.py b/visidata/canvas.py
--- a/visidata/canvas.py
+++ b/visidata/canvas.py
@@ -482,7 +482,15 @@ def resetBounds(self):
                     if ymin is None or y < ymin: ymin = y
                     if xmax is None or x > xmax: xmax = x
                     if ymax is None or y > ymax: ymax = y
-            self.canvasBox = BoundingBox(float(xmin or 0), float(ymin or 0), float(xmax or 0)+1, float(ymax or 0)+1)
+            xmin = xmin or 0
+            xmax = xmax or 0
+            ymin = ymin or 0
+            ymax = ymax or 0
+            if xmin == xmax:
+                xmax += 1
+            if ymin == ymax:
+                ymax += 1
+            self.canvasBox = BoundingBox(float(xmin), float(ymin), float(xmax), float(ymax))
 
         if not self.visibleBox:
             # initialize minx/miny, but w/h must be set first to center properly


Test patch:
Repository: saulpw/visidata

Issue:
[capture-col] When nothing is captured, "IndexError: list index out of range" is thrown
_Minor user exp bug_

**Small description**
When using `capture-col` on a column and the regex does not capture any values, the below error is thrown. No new column is created (as expected)

**Expected result**
Just to show the "no columns to add" error. Preferably, change the error to something more informative such as "no matches found, didn't add column"

**Actual result with screenshot**
```
Traceback (most recent call last):
  File "/Users/geekscrapy7/lib/python3.7/site-packages/visidata/threads.py", line 200, in _toplevelTryFunc
    t.status = func(*args, **kwargs)
  File "/Users/geekscrapy7/lib/python3.7/site-packages/visidata/features/regex.py", line 77, in addRegexColumns
    vs.addColumnAtCursor(*cols.values())
  File "/Users/geekscrapy7/lib/python3.7/site-packages/visidata/sheets.py", line 473, in addColumnAtCursor
    firstnewcol = [c for c in cols if not c.hidden][0]
IndexError: list index out of range
```
I presume this error is from an insufficient catch of an exception and it is attempting to create the column even though there are no groups to show.

**Steps to reproduce with sample data and a .vd**
`# echo -e 'helloooo\nthere' | vd -f txt --config=/dev/null --visidata_dir=/dev/null`

```tsv
sheet	col	row	longname	input	keystrokes	comment
-	text		capture-col	(ABC)	;	add new column from capture groups of regex; requires example row
```

**Additional context**
v2.12dev


Discussion:
Good improvement! I suggest that the error message should match the case when the user forgot to specify a capture group. I often forget to, and I'm probably not the only one.

Resolving patch:
diff --git a/visidata/features/regex.py b/visidata/features/regex.py
--- a/visidata/features/regex.py
+++ b/visidata/features/regex.py
@@ -24,6 +24,8 @@ def makeRegexSplitter(vd, regex, origcol):
 
 @VisiData.api
 def makeRegexMatcher(vd, regex, origcol):
+    if not regex.groups:
+        vd.fail('specify a capture group')  #1778
     def _regexMatcher(row):
         m = regex.search(origcol.getDisplayValue(row))
         if m:
@@ -74,6 +76,10 @@ def addRegexColumns(vs, regexMaker, origcol, regexstr):
         else:
             raise TypeError("addRegexColumns() expects a dict, list, or tuple from regexMaker, but got a "+type(m).__name__)
 
+    if not cols:
+        vd.warning("no regex matches found, didn't add column")
+        return
+
     vs.addColumnAtCursor(*cols.values())
 
 


Test patch:
Repository: saulpw/visidata

Issue:
Feature request: Provide command `reopen-closed-sheet` like Chrome's Ctrl-Shift-T
It often happens that I close a sheet by mistake and immediately curse because I've already done a bunch of manipulations on this tab that I'll now have to repeat. (I know I can guard a sheet but that doesn't fit my workflow.)

I wish VisiData had a feature like Chrome's Ctrl-Shift-T. Basically a command that just opens that sheet that was closed most recently, with all the manipulations that I've done to it. 

Of course, this command is challenging because it means VisiData has to keep this information around. Maybe limit it to last 3 sheets.


Discussion:
Actually there is already `g Shift+S` which has a list of all sheets ever made, and you can select the closed sheet from there to reopen it.  (`Shift+Q` will quit it without recourse, however.)  It wouldn't be too difficult to make this command.  The question is, what to bind it to?  `zq`?  `g Shift+U`?
Good to know about `g Shift+S`! This almost solves the problem for me. I do notice that it's sorted by name though. I'm interested in the most recently closed sheet, so this might be hard to find if I have more than a dozen sheets there. It would make sense to me if these were sorted according to datetime of closing. 

I figure that maybe I can use the `shortcut` column as a proxy for last closing, which might work. One weird thing I found is that when sorting by that column, either descending or ascending, it stays in the ascending order of 1-2-3-4. Setting the column as int or float and then trying to sort again makes no difference.

Regarding a key binding, I don't know. Your intuition is probably better than mine. I'm okay with both of the options you suggested.
So I think this command is just:

```
BaseSheet.addCommand('zq', 'open-last-closed', 'push(allSheets[-1])'.  'reopen most recently closed sheet')
```

Can you try this out and see how it feels?  I'm not sure about `zq`, it felt kind of awkward, but please try `zU` and `gU` and see how they all feel.  Submit a PR if you like one of them (and include `vd.addMenuItems('...') with where it goes in the menu scheme).

Resolving patch:
diff --git a/visidata/_open.py b/visidata/_open.py
--- a/visidata/_open.py
+++ b/visidata/_open.py
@@ -185,7 +185,9 @@ def loadInternalSheet(vd, cls, p, **kwargs):
 
 BaseSheet.addCommand('o', 'open-file', 'vd.push(openSource(inputFilename("open: "), create=True))', 'Open file or URL')
 TableSheet.addCommand('zo', 'open-cell-file', 'vd.push(openSource(cursorDisplay) or fail(f"file {cursorDisplay} does not exist"))', 'Open file or URL from path in current cell')
+BaseSheet.addCommand('gU', 'undo-last-quit', 'push(allSheets[-1])', 'reopen most recently closed sheet')
 
 vd.addMenuItems('''
     File > Open file/url > open-file
+    File > Reopen last closed > undo-last-quit
 ''')


Test patch:
Repository: saulpw/visidata

Issue:
undo `zd` cannot remove `[M]`
**Steps to reproduce with sample data and a .vd**
```sh
echo 'a\tb\tc\n1\t2\te' > test.tsv
vd -N test.tsv
```
Press `zd` to delete one cell then use `U` to undo. Now the table is same as original table. However, `sheet.modifiedStatus` still be `[M]`.

**Expected result**

`sheet.modifiedStatus` still be empty string.


Discussion:
What a cool bug!

This actually applies to any command that calls `setValue`. `setValue` sets `hasBeenModified` to `True`. 

The undo for `hasBeenModified` gets created after the undo for `zd`. We undo the `hasBeenModified` change, and it gets set back to `False`.

But then, the undo for `setValue`/`zd` also call `setValue`, and then `hasBeenModified` gets set back to `True`.

I encountered this before. The `undo` for `delete-row` called `add-row` which has its own undo schematics that mucked with things. In that case, the fix was pretty tricky, and it might be for this one as well. I am going to look into it!

Resolving patch:
diff --git a/visidata/column.py b/visidata/column.py
--- a/visidata/column.py
+++ b/visidata/column.py
@@ -380,13 +380,14 @@ def putValue(self, row, val):
         if self.setter:
             return self.setter(self, row, val)
 
-    def setValue(self, row, val):
+    def setValue(self, row, val, setModified=True):
         'Change value for *row* in this column to *val*.  Call ``putValue`` immediately if not a deferred column (added to deferred parent at load-time); otherwise cache until later ``putChanges``.  Caller must add undo function.'
         if self.defer:
             self.cellChanged(row, val)
         else:
             self.putValue(row, val)
-        self.sheet.setModified()
+        if setModified:  #1800
+            self.sheet.setModified()
 
     def setValueSafe(self, row, value):
         'setValue and ignore exceptions.'
diff --git a/visidata/undo.py b/visidata/undo.py
--- a/visidata/undo.py
+++ b/visidata/undo.py
@@ -99,7 +99,7 @@ def addUndoSetValues(vd, cols, rows):
     oldvals = [(c, r, c.getValue(r)) for c,r in itertools.product(cols, vd.Progress(rows, gerund='doing'))]
     def _undo():
         for c, r, v in oldvals:
-            c.setValue(r, v)
+            c.setValue(r, v, setModified=False)
     vd.addUndo(_undo)
 
 @VisiData.api


Test patch:
Repository: saulpw/visidata

Issue:
resize-cols-input missing from Column -> Resize menu
**Small description**
resize-cols-input is missing from the Column -> Resize menu

**Expected result**
when I go to the menu under Column -> Resize, I expect resize-cols-input (gz_) to be given as an option

**Additional context**
checked against v2.11

(Submitting pull request to fix this now.)




Discussion:

Resolving patch:
diff --git a/visidata/features/layout.py b/visidata/features/layout.py
--- a/visidata/features/layout.py
+++ b/visidata/features/layout.py
@@ -50,4 +50,5 @@ def unhide_cols(vd, cols, rows):
     Column > Resize > current column to max > resize-col-max
     Column > Resize > current column to input > resize-col-input
     Column > Resize > all columns max > resize-cols-max
+    Column > Resize > all columns to input > resize-cols-input
 ''')


Test patch:
Repository: saulpw/visidata

Issue:
Keystroke ] not detected on Windows
In Powershell and cmd.exe I encountered that sorting didn't work in both orders. The `[` shortcut was detected and had its effect, but the `]` didn't. I narrowed it down to a problem with `windows-curses`, and in turn with its dependency `PDCurses`: https://github.com/zephyrproject-rtos/windows-curses/issues/41

Here's my plan on how to address it. I hope I'll get around to it somewhere next week.
- [ ] Improve the mapping in `PDCurses` and submit a pull request
- [ ] Bump the git submodule in `windows-curses` to the `PDCurses` version that has the fix and ask/wait for a release of this package
- [ ] Address the issue in this repository, perhaps by pinning `windows-curses` to a version of at least the newly released package.

I'm making this issue here just to document it and track progress. If you're reading this because you have this issue, I would recommend using WSL instead. (WSL is not an option for me unfortunately).

I didn't include the `.vd`-file to reproduce this issue. The simplest way to reproduce it is to get a Windows computer, run `visidata` from Powershell or cmd.exe and sort any column by pressing `]`.


Discussion:
Thanks for the information, @bartbroere.  This is a fascinating bug.  Is this perhaps because of a non-US keyboard?  I'm surprised (though perhaps I shouldn't be) that Windows is having a problem entering a basic ASCII character in 2023.
Yes, mine is not US standard indeed. I encountered the issue both with the NLD DV (Dutch Dvorak) layout and the NLD INTL (Dutch QWERTY) layout.

The Windows API actually is quite nice, it has [virtual key codes](https://learn.microsoft.com/en-us/windows/win32/inputdev/virtual-key-codes) so you don't have to deal with the different layouts yourself. (If I understand correctly). The `PDCurses` project does seem to use this functionality.

I'll keep the updates coming here as I discover more. It's fascinating indeed!

Edit: Fixed wrong statement

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -23,7 +23,7 @@
       py_modules=['visidata'],
       install_requires=[
           'python-dateutil',
-          'windows-curses; platform_system == "Windows"',
+          'windows-curses<2.3.1; platform_system == "Windows"',  #1841
           'importlib-metadata >= 3.6',
       ],
       packages=['visidata', 'visidata.loaders', 'visidata.vendor', 'visidata.tests', 'visidata.ddw', 'visidata.man', 'visidata.themes', 'visidata.features', 'visidata.experimental', 'visidata.apps', 'visidata.apps.vgit', 'visidata.apps.vdsql', 'visidata.desktop'],


Test patch:
Repository: saulpw/visidata

Issue:
Current HEAD zsh-completion.py needs option_aliases update
**Small description**

`option_aliases` was removed in ce497f444db6d2f3fc0b8309f5ca839196c33c8b but is still referred to in the zsh completion code.

https://github.com/saulpw/visidata/blob/34808745232e798b0f25e893bb444fc9f3c034eb/dev/zsh-completion.py#L11C41-L11C41

I think the script needs a slight rejig to use the (present) `vd` import instead.

I wonder whether this can be included in future CI?

**Expected result**

The command succeeds.

**Actual result**

```
       > /build/visidata-src
       > Traceback (most recent call last):
       >   File "/build/visidata-src/dev/zsh-completion.py", line 11, in <module>
       >     from visidata.main import option_aliases
       > ImportError: cannot import name 'option_aliases' from 'visidata.main' (/build/visidata-src/visidata/main.py)
```

**Steps to reproduce**

```
python dev/zsh-completion.py
```

**Additional context**
~~Please include the version of VisiData and Python.~~

https://github.com/saulpw/visidata/tree/34808745232e798b0f25e893bb444fc9f3c034eb but I listed the commit above that causes the breakage — I suspect this is a two minute fix for somebody familiar with the codebase, though not me. I can help with extending CI, though it might just be a case of adding

```yaml
    - name: Ensure VisiData can create completions
      run: python dev/zsh-completion.py
```

(I guess you might want to run a linter, instead.)


Discussion:
I'm also seeing failing tests, but my build tooling isn't letting me past this so I can't check whether they're still broken on develop: `describe-through`, `histogram`, `petsdiet` and `remove-errors`.

```
AttributeError: 'NoneType' object has no attribute 'erase'
```
Could you please open a PR with the fixes you mention?

The zsh plugin has not been added by the maintainers, and we are not in the best place to update it ourselves. We appreciate it if the community can help out.
Ah, I see, the problem was an API change on our end. I will take a look before the next release.

Resolving patch:
diff --git a/dev/zsh-completion.py b/dev/zsh-completion.py
--- a/dev/zsh-completion.py
+++ b/dev/zsh-completion.py
@@ -8,7 +8,6 @@
 
 sys.path.insert(0, dirn(dirn((os.path.abspath(__file__)))))
 from visidata import vd
-from visidata.main import option_aliases
 
 ZSH_COMPLETION_FILE = "_visidata"
 ZSH_COMPLETION_TEMPLATE = "dev/zsh-completion.in"
@@ -18,7 +17,9 @@
 
 def generate_completion(opt):
     prefix = "--" + opt.name
-    shortnames = [key for key, value in option_aliases.items() if value[0] == opt.name]
+    shortnames = [
+        key for key, value in vd.option_aliases.items() if value[0] == opt.name
+    ]
     if len(shortnames):
         if len(shortnames[0]) == 1:
             shortname = "-" + shortnames[0]


Test patch: