internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: saulpw/visidata

Issue:
[pandas/html] Stack trace when loading html data with pandas
**Small description**
Stack trace when loading html data with pandas

**Expected result**
Return an IndexSheet of PandaSheets of the HTML tables found in an html file.

**Actual result with screenshot**

https://asciinema.org/a/650fulFvz3V9FXwJVkCkuDpeP

```
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/visidata/threads.py", line 200, in _toplevelTryFunc                         
    t.status = func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/visidata/loaders/_pandas.py", line 171, in reload
    df = df.reset_index(drop=True)
AttributeError: 'list' object has no attribute 'reset_index'
```

**Steps to reproduce with sample data and a .vd**

```
vd -f pandas sample_data/sunshinelist.html
```

Note: Visidata code expects the read_* functions return a DataFrame.  The read_html returns a list of DataFrames:
https://pandas.pydata.org/docs/reference/api/pandas.read_html.html
 

**Additional context**
Please include the version of VisiData and Python.
Latest develop version.  3.9.2



Discussion:

Resolving patch:
diff --git a/visidata/loaders/_pandas.py b/visidata/loaders/_pandas.py
--- a/visidata/loaders/_pandas.py
+++ b/visidata/loaders/_pandas.py
@@ -158,6 +158,12 @@ def reload(self):
                 readfunc = getattr(pd, 'read_'+filetype) or vd.error('no pandas.read_'+filetype)
             # readfunc() handles binary and text open()
             df = readfunc(self.source, **options.getall('pandas_'+filetype+'_'))
+            # some read methods (html, for example) return a list of dataframes
+            if isinstance(df, list):
+                for idx, inner_df in enumerate(df[1:], start=1):
+                    vd.push(PandasSheet(f'{self.name}[{idx}]', source=inner_df))
+                df = df[0]
+                self.name += '[0]'
             if (filetype == 'pickle') and not isinstance(df, pd.DataFrame):
                 vd.fail('pandas loader can only unpickle dataframes')
         else:


Test patch:
Repository: saulpw/visidata

Issue:
Bug: cannot read .vds with expression column
**Small description**

Visidata cannot read back sheet(s) it saved as `.vds` if they contain an
expression column.

"KeyError: 'ExprColumn'" shows as error, resulting in a partial read.


**Expected result**

It should be able to read those files.


**Actual result with ~~screenshot~~ stacktrace**

```
Traceback (most recent call last):
  File "/nix/store/z4xjb4j8i73894r2wqjvlnps9j60rjr0-visidata-2.11/lib/python3.10/site-packages/visidata/threads.py", line 198, in _toplevelTryFunc
    t.status = func(*args, **kwargs)
  File "/nix/store/z4xjb4j8i73894r2wqjvlnps9j60rjr0-visidata-2.11/lib/python3.10/site-packages/visidata/pyobj.py", line 26, in reload
    for r in self.iterload():
  File "/nix/store/z4xjb4j8i73894r2wqjvlnps9j60rjr0-visidata-2.11/lib/python3.10/site-packages/visidata/loaders/vds.py", line 76, in iterload
    c = globals()[classname](d.pop('name'), sheet=self)
KeyError: 'ExprColumn'
```


**Steps to reproduce with sample data and a .vd**

Create and save some test sheet with an expr column with this `cmdlog.vdj`:

```
#!vd -p
{"col": "", "row": "", "longname": "open-new", "input": "", "keystrokes": "Shift+A", "comment": "Open new empty sheet"}
{"sheet": "unnamed", "col": "A", "row": "", "longname": "type-int", "input": "", "keystrokes": "#", "comment": "set type of current column to int"}
{"sheet": "unnamed", "col": "", "row": "", "longname": "add-row", "input": "", "keystrokes": "a", "comment": "append a blank row"}
{"sheet": "unnamed", "col": "A", "row": "0", "longname": "edit-cell", "input": "2", "keystrokes": "e", "comment": "edit contents of current cell"}
{"sheet": "unnamed", "col": "A", "row": "", "longname": "addcol-expr", "input": "A*2", "keystrokes": "=", "comment": "create new column from Python expression, with column names as variables"}
{"sheet": "unnamed", "col": "", "row": "", "longname": "save-sheet", "input": "sheet.vds", "keystrokes": "Ctrl+S", "comment": "save current sheet to filename in format determined by extension (default .tsv)"}
```

This produces `sheet.vds` as follows, which seems valid:

```
#{"name": "unnamed"}
#{"name": "A", "width": 4, "height": 1, "expr": null, "keycol": 0, "formatter": "", "fmtstr": "", "voffset": 0, "hoffset": 0, "aggstr": "", "type": "int", "col": "Column"}
#{"name": "A*2", "width": 5, "height": 1, "expr": "A*2", "keycol": 0, "formatter": "", "fmtstr": "", "voffset": 0, "hoffset": 0, "aggstr": "", "type": "", "col": "ExprColumn"}
{"A": 2, "A*2": 4}
```

Quit visidata and open that file again with `vd sheet.vds`,
and observe the loading error.


**Additional context**

- visidata v2.11
- python 3.10.12



Discussion:

Resolving patch:
diff --git a/visidata/loaders/vds.py b/visidata/loaders/vds.py
--- a/visidata/loaders/vds.py
+++ b/visidata/loaders/vds.py
@@ -2,7 +2,7 @@
 
 import json
 
-from visidata import VisiData, JsonSheet, Progress, IndexSheet, SettableColumn, ItemColumn
+from visidata import VisiData, JsonSheet, Progress, IndexSheet, SettableColumn, ItemColumn, ExprColumn
 
 
 NL='\n'


Test patch:
Repository: saulpw/visidata

Issue:
[parquet] can't load parquet directory anymore: `IsADirectoryError`
**Small description**

Hi @saulpw @anjakefala @takacsd - it seems that forcing opening the path as file with `.open()` - introduced with #2133 - breaks the use case where the multiple parquet files are stored in a directory, and this directory is then read by visidata. This is common with Hive partitioning or when working with spark. A simple fix would be to check if the path is a directory with `os.path.is_dir()` and then retaining old behavior of passing it as a string to `read_table()`. If it is not an existing directory, we move to the new way of opening as a binary buffer.

I have already added this workaround to my clone of visidata, and it fixes my issue, but maybe you have some better ideas how to handle it instead of `if-else` statement in the `ParquetSheet`.

**Expected result**

```bash
vd -f parquet parquet_dir
```
should load a parquet into visidata

**Actual result with screenshot**
![image](https://github.com/saulpw/visidata/assets/9008816/f262c403-704e-47f0-8994-68bf214b3db6)

**Additional context**

```bash
# freshest develop
visidata@9fd728b72c115e50e99c24b455caaf020381b48e

pyarrow==12.0.0
python 3.10.2
```



Discussion:

Resolving patch:
diff --git a/visidata/loaders/parquet.py b/visidata/loaders/parquet.py
--- a/visidata/loaders/parquet.py
+++ b/visidata/loaders/parquet.py
@@ -23,8 +23,11 @@ def iterload(self):
         pq = vd.importExternal("pyarrow.parquet", "pyarrow")
         from visidata.loaders.arrow import arrow_to_vdtype
 
-        with self.source.open('rb') as f:
-            self.tbl = pq.read_table(f)
+        if self.source.is_dir():
+            self.tbl = pq.read_table(str(self.source))
+        else: 
+            with self.source.open('rb') as f:
+                self.tbl = pq.read_table(f)
 
         self.columns = []
         for colname, col in zip(self.tbl.column_names, self.tbl.columns):


Test patch:
Repository: saulpw/visidata

Issue:
inputMultiple() displays a malformed empty status box
**Small description**
`search-keys` and `search-col` displays an empty box while waiting for the user to input a regex at the prompt. The box is shown where the usual status messages are shown.

**Expected result**
No box should be displayed.

**Actual result with screenshot**
![sidebar-inputMultiple](https://github.com/saulpw/visidata/assets/67946319/1b068536-bd38-4953-bfc0-640920380ded)

For comparison, here is what the normal status messages look like:
![sidebar-normal](https://github.com/saulpw/visidata/assets/67946319/de49d377-aaab-4b1b-9ac3-56906b7c53a3)

**Steps to reproduce with sample data and a .vd**
Open any sheet and hit `r` or `/`.

**Additional context**
saul.pw/VisiData v3.0dev
Python 3.10.12
Ubuntu 22.04.3


Discussion:
Hm, these are help panels which should be installed via pip now.  Can you try uninstalling and reinstalling?

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -29,7 +29,7 @@
       ],
       packages=['visidata', 'visidata.loaders', 'visidata.vendor', 'visidata.tests', 'visidata.ddw', 'visidata.man', 'visidata.themes', 'visidata.features', 'visidata.experimental', 'visidata.apps', 'visidata.apps.vgit', 'visidata.apps.vdsql', 'visidata.desktop'],
       data_files=[('share/man/man1', ['visidata/man/vd.1', 'visidata/man/visidata.1']), ('share/applications', ['visidata/desktop/visidata.desktop'])],
-      package_data={'visidata.man': ['vd.1', 'vd.txt'], 'visidata.ddw': ['input.ddw'], 'visidata.tests': ['sample.tsv'], 'visidata.desktop': ['visidata.desktop']},
+      package_data={'visidata.man': ['vd.1', 'vd.txt'], 'visidata.ddw': ['input.ddw', 'regex.ddw'], 'visidata.tests': ['sample.tsv'], 'visidata.desktop': ['visidata.desktop']},
       license='GPLv3',
       classifiers=[
           'Development Status :: 5 - Production/Stable',


Test patch:
Repository: saulpw/visidata

Issue:
save as csv actually saves tsv when file existing file extension is CSV, i.e.  uppercase - PR available #2269
**Small description**

Slightly hperbolic ;-) Corruption of file format, opening a csv but saving as csv results in TSV data in a file named csv

**Expected result**

CSV, not TSV

**Actual result with screenshot**

If you get an unexpected error, please include the full stack trace that you get with `Ctrl-E`.
No error, contents:

    header1	header2
    1	one
    2	two


**Steps to reproduce with sample data and a .vd**

Datafile, called bug.CSV

    header1,header2
    1,one
    2,two


1. open data file, MUST have uppercase CSV on end (works fine for lower). E.g., `visidata bug.CSV`
2. save (ctrl-s)
3. hit enter to accept current filename
4. hit `y` to overwrite
5. Display will say saving TSV
6. sanity check file contents

First try reproducing without any user configuration by using the flag `-N`.
e.g. `echo "abc" | vd -f txt -N`

Please attach the commandlog (saved with `Ctrl-D`) to show the steps that led to the issue.
See [here](http://visidata.org/docs/save-restore/) for more details.

**Additional context**
Please include the version of VisiData and Python.

Windows:

    (py311csv) C:\code\py>python
    Python 3.11.3 (tags/v3.11.3:f3909b8, Apr  4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> ^Z


    (py311csv) C:\code\py>visidata --version
    saul.pw/VisiData v3.0.2




Discussion:
    visidata --save-filetype=csv bug.CSV

Works fine, so likely autodetect code.

Resolving patch:
diff --git a/visidata/save.py b/visidata/save.py
--- a/visidata/save.py
+++ b/visidata/save.py
@@ -110,7 +110,7 @@ def saveSheets(vd, givenpath, *vsheets, confirm_overwrite=True):
         vd.warning('no sheets to save')
         return
 
-    filetypes = [givenpath.ext, vd.options.save_filetype]
+    filetypes = [givenpath.ext.lower(), vd.options.save_filetype.lower()]
 
     vd.clearCaches()
 


Test patch:
Repository: saulpw/visidata

Issue:
[cmdlog] sheets created with no keypresses show errors 
**Small description**
clicking on `dir_hidden` in the DirSheet guide raises an exception

**Actual result with screenshot**
```
File "/home/midichef/.local/lib/python3.10/site-packages/visidata/statusbar.py", line 56, in sheetlist
if len(vs.shortcut) == 1:
TypeError: object of type 'NoneType' has no len()
```

**Steps to reproduce with sample data and a .vd**
`vd .`, navigate to `filename` column to bring the DirSheet guide up, click on `dir_hidden`

**Additional context**
visidata 3.1dev

It looks like vs.shortcut is `None` because some code for `shortcut()` is obsolete, where it checks `cmdlog.rows[0].keystrokes`:
https://github.com/saulpw/visidata/blob/aa9d2615f3b2773001cf75a1b24219903a91c1bb/visidata/cmdlog.py#L415


Discussion:
Okay I've got a clearer idea of exactly when this happens. It's when you create a sheet without using keystrokes. Like a command started by `exec-command`, or a help menubar entry, or clicking a statusbar link. The sheet has to have `precious == False`, like a HelpSheet or OptionsSheet. So. running `exec-longname` and then choosing `help-commands`.
https://github.com/saulpw/visidata/blob/c702c3512ba81584ec8f2dad30e7dc75eba1e943/visidata/cmdlog.py#L412
Because the sheet isn't precious, it can't be found in `vd.allSheets`, so the command log is searched for keystrokes. But there are no keystrokes because the action that triggered the command wasn't a keypress.
As a secondary issue, I wonder if
https://github.com/saulpw/visidata/blob/c702c3512ba81584ec8f2dad30e7dc75eba1e943/visidata/cmdlog.py#L417
has a remaining problem, when it uses `rows[0]`. It assumes the first row of the command log is the one that created the sheet. But they're usually `set-options` commands. (This same assumption caused a problem with undo in https://github.com/saulpw/visidata/pull/2244.) It seems like that may cause an error.

I haven't yet found a case it does. But before I dig deeper, what is this shortcut code for? What's an example where a keystroke from the command log can be thought of as a shortcut?
> What is this shortcut code for? What's an example where a keystroke from the command log can be thought of as a shortcut?

The shortcut shown on the status bar before the sheetname: `1›sample`.  For regular sheets, it's assigned; the intention is that if you opened a non-precious sheet, the "shortcut" would be the key you used to open it, like `C` for the columns sheet or `O` for the options sheet.  But this usage was disabled (when `Shift+C` just did not look right and I wasn't sure about it anyway), so these days the shortcut is only meaningful for precious sheets.

Thanks for looking into all this, by the way.  I know this code can get pretty convoluted over time.

Resolving patch:
diff --git a/visidata/cmdlog.py b/visidata/cmdlog.py
--- a/visidata/cmdlog.py
+++ b/visidata/cmdlog.py
@@ -414,7 +414,7 @@ def shortcut(self):
         pass
 
     try:
-        return self.cmdlog_sheet.rows[0].keystrokes
+        return self.cmdlog_sheet.rows[0].keystrokes or ''  #2293
     except Exception:
         pass
 


Test patch:
Repository: saulpw/visidata

Issue:
[csv] load % not working for csv on disk
When loading a larger .csv file from disk, the loading status bar says `0%` the entire time.


Discussion:

Resolving patch:
diff --git a/visidata/loaders/csv.py b/visidata/loaders/csv.py
--- a/visidata/loaders/csv.py
+++ b/visidata/loaders/csv.py
@@ -71,9 +71,10 @@ def save_csv(vd, p, sheet):
         if ''.join(colnames):
             cw.writerow(colnames)
 
-        with Progress(gerund='saving'):
+        with Progress(gerund='saving', total=sheet.nRows) as prog:
             for dispvals in sheet.iterdispvals(format=True):
                 cw.writerow(dispvals.values())
+                prog.addProgress(1)
 
 CsvSheet.options.regex_skip = '^#.*'
 
diff --git a/visidata/path.py b/visidata/path.py
--- a/visidata/path.py
+++ b/visidata/path.py
@@ -111,6 +111,7 @@ def __init__(self, path, fp, mode='r', **kwargs):
 
         # track Progress on original fp
         self.fp_orig_read = self.fp.read
+        self.fp_orig_readline = self.fp.readline
         self.fp_orig_close = self.fp.close
 
         self.fp.read = self.read
@@ -132,6 +133,12 @@ def read(self, size=-1):
                 self.prog.addProgress(len(r))
         return r
 
+    def readline(self, size=-1):
+        r = self.fp_orig_readline(size)
+        if self.prog:
+            self.prog.addProgress(len(r))
+        return r
+
     def __getattr__(self, k):
         return getattr(self.fp, k)
 


Test patch:
Repository: saulpw/visidata

Issue:
Python error after pressing SPACE to pull up command-name input
**Small description**
The first time I press SPACE on a sheet, I often get the following Python error. It usually succeeds on the second press.

**Expected result**
Open command prompt.

**Actual result with screenshot**
![Screenshot from 2024-03-03 15-07-48](https://github.com/saulpw/visidata/assets/6864226/e9245111-df2d-4958-97be-66c5abcfb9ec)
[errors_recent.txt](https://github.com/saulpw/visidata/files/14471969/errors_recent.txt)

**Steps to reproduce with sample data and a .vd**
Open any sheet and press SPACE.
It may take a few attempts. It seems to only happen the first time it is entered on a sheet, and only some of the time.

**Additional context**
Python 3.8.0
VisiData 3.0.2



Discussion:
This is another error that also very frequently appears instead of the previous error.
[errors_recent.txt](https://github.com/saulpw/visidata/files/14472478/errors_recent.txt)

Also, many commands such as "guard-sheet" do not populate in the list or execute on one of my computers, even though they are still included in the command list. I've tried uninstalling and reinstalling Visidata, updating pip and Python to the latest version and reinstalling Visidata, made sure to delete the original files from ~/.visidata and run without an rc file, and yet these commands seem not to be available on that machine.

My fix for now on that machine is to downgrade to 2.11. 3.0.2 works well enough on my other computer, despite this intermittent error.
> Also, many commands such as "guard-sheet" do not populate in the list or execute on one of my computers, even though they are still included in the command list. I've tried uninstalling and reinstalling Visidata, updating pip and Python to the latest version and reinstalling Visidata, made sure to delete the original files from ~/.visidata and run without an rc file, and yet these commands seem not to be available on that machine.

This is a seperate issue than the original one you posted! Could you please open a new issue with a reproduction?

Resolving patch:
diff --git a/visidata/features/cmdpalette.py b/visidata/features/cmdpalette.py
--- a/visidata/features/cmdpalette.py
+++ b/visidata/features/cmdpalette.py
@@ -167,7 +167,7 @@ def inputLongname(sheet):
     prompt = 'command name: '
     # get set of commands possible in the sheet
     this_sheets_help = HelpSheet('', source=sheet)
-    this_sheets_help.ensureLoaded()
+    vd.sync(this_sheets_help.ensureLoaded())
 
     def _fmt_cmdpal_summary(match, row, trigger_key):
         keystrokes = this_sheets_help.revbinds.get(row.longname, [None])[0] or ' '


Test patch:
Repository: saulpw/visidata

Issue:
unable to edit nested json value
**Small description**

Attempting to edit a nested value in JSON results in `NameError: name 'setitem' is not defined`

**Expected result**

Pressing `Enter` in edit mode for a value changes the value

**Actual result with screenshot**

![image](https://github.com/saulpw/visidata/assets/4935266/af19ccae-9eea-4abd-b623-31b95a6d143d)

[errors_recent.txt](https://github.com/saulpw/visidata/files/15048809/errors_recent.txt)

**Steps to reproduce with sample data and a .vd**

1. Open JSON file such as:
    ```json
    {
      "key1": {
        "key2": {
          "value1": 1,
          "value2": 2
        }
      }
    }
    ```
2. Navigate to the value for `value1`
3. Press `e`
4. Enter a new value
5. Press `Enter`

[cmd-log.txt](https://github.com/saulpw/visidata/files/15048823/cmd-log.txt)

**Additional context**

```shell
$ vd --version
saul.pw/VisiData v3.0.2

$ python --version
Python 3.10.12
```


Discussion:

Resolving patch:
diff --git a/visidata/pyobj.py b/visidata/pyobj.py
--- a/visidata/pyobj.py
+++ b/visidata/pyobj.py
@@ -4,7 +4,7 @@
 import numbers
 
 from visidata import vd, asyncthread, ENTER, deduceType
-from visidata import Sheet, Column, VisiData, ColumnItem, TableSheet, BaseSheet, Progress, ColumnAttr, SuspendCurses, TextSheet
+from visidata import Sheet, Column, VisiData, ColumnItem, TableSheet, BaseSheet, Progress, ColumnAttr, SuspendCurses, TextSheet, setitem
 import visidata
 
 vd.option('visibility', 0, 'visibility level')


Test patch:
Repository: saulpw/visidata

Issue:
`history` parameter of input() is appears ignored
**Small description**

I think the `history` parameter of the `input` function is unused and overwritten here:
https://github.com/saulpw/visidata/blob/3ad7a3d0c1475ff53bf31481506b9a748b48ac7c/visidata/_input.py#L501

**Expected result**

That `history` would be honored. I believe it is not.

**Actual result with screenshot**

(no screenshot, this shows up when you call this function and use the arrow keys to go back into the history.)

**Steps to reproduce with sample data and a .vd**

Try this in your `.visidatarc`:
```
VIEWER_HISTORY = []


def guess_program(filename: str) -> str:
    basename, ext = path.splitext(filename)
    ext = ext.lower()
    with open("/tmp/log", "w") as f:
        print(ext, file=f)
    if ext == '.pdf':
        return 'evince'
    elif ext == '.jpg':
        return 'geeqie'


@visidata.VisiData.api
def launchViewer(vd, *args):
    "Launch $EDITOR with *args* as arguments."
    global VIEWER_HISTORY

    filename = str(args[0])
    if not path.exists(filename):
        vd.fail(f"Value '{filename}' is not an existing filename.")

    program = guess_program(filename)
    if program and (not VIEWER_HISTORY or VIEWER_HISTORY[0] != program):
        VIEWER_HISTORY.append(program)

    viewer = vd.input(
        "viewer: ",
        history=VIEWER_HISTORY,
        defaultLast=bool(VIEWER_HISTORY),
    )
    if not viewer.strip():
        vd.fail("Viewer not provided")
    elif viewer not in VIEWER_HISTORY:
        VIEWER_HISTORY.insert(0, viewer)

    args = viewer.split() + list(args)
    with SuspendCurses():
        # subprocess.call(args)                                                 
        cproc = subprocess.Popen(args)


DirSheet.addCommand(
    "Alt+!",
    "sysopen-row-shell-command",
    "launchViewer(cursorRow)",
    "open current file in external program (input manually)",
)

```


**Additional context**

git reflog: 3ad7a3d0c1475ff53bf31481506b9a748b48ac7c
date: 2024-04-28 
python: 3.12.1


Thank you thank you!




Discussion:

Resolving patch:
diff --git a/visidata/_input.py b/visidata/_input.py
--- a/visidata/_input.py
+++ b/visidata/_input.py
@@ -528,7 +528,8 @@ def input(vd, prompt, type=None, defaultLast=False, history=[], dy=0, attr=None,
             import getpass
             return getpass.getpass(prompt)
 
-    history = list(vd.inputHistory.setdefault(type, {}).keys())
+    if not history:
+        history = list(vd.inputHistory.setdefault(type, {}).keys())
 
     y = sheet.windowHeight-dy-1
     promptlen = dispwidth(prompt)


Test patch:
Repository: mytardis/mytardis

Issue:
Stream single files
Change the single file download API endpoint to stream file contents



Discussion:

Resolving patch:
diff --git a/tardis/tardis_portal/api.py b/tardis/tardis_portal/api.py
--- a/tardis/tardis_portal/api.py
+++ b/tardis/tardis_portal/api.py
@@ -13,7 +13,8 @@
 from django.contrib.auth.models import User
 from django.contrib.auth.models import Group
 from django.core.servers.basehttp import FileWrapper
-from django.http import HttpResponse, HttpResponseForbidden
+from django.http import HttpResponse, HttpResponseForbidden, \
+    StreamingHttpResponse
 
 from tardis.tardis_portal.auth.decorators import \
     get_accessible_datafiles_for_user
@@ -900,7 +901,8 @@ def download_file(self, request, **kwargs):
             self.build_bundle(obj=file_record, request=request))
         file_object = file_record.get_file()
         wrapper = FileWrapper(file_object)
-        response = HttpResponse(wrapper, content_type=file_record.mimetype)
+        response = StreamingHttpResponse(
+            wrapper, content_type=file_record.mimetype)
         response['Content-Length'] = file_record.size
         response['Content-Disposition'] = 'attachment; filename="%s"' % \
                                           file_record.filename


Test patch: