Repository: spyder-ide/spyder

Issue:
How to view numbers in a dataframe with thousands separator in spyder variable explorer?
<!--- **PLEASE READ:** When submitting here, please ensure you've completed the following checklist and checked the boxes to confirm. Issue reports without it may be closed. Thanks! --->

### Issue Report Checklist

* [ ] Searched the [issues page](https://github.com/spyder-ide/spyder/issues?q=is%3Aissue) for similar reports
* [ ] Read the relevant sections of the [Spyder Troubleshooting Guide](https://github.com/spyder-ide/spyder/wiki/Troubleshooting-Guide-and-FAQ) and followed its advice
* [ ] Reproduced the issue after updating with ``conda update spyder`` (or ``pip``, if not using Anaconda)
* [ ] Could not reproduce inside ``jupyter qtconsole`` (if console-related)
* [ ] Tried basic troubleshooting (if a bug/error)
    * [ ] Restarted Spyder
    * [ ] Reset preferences with ``spyder --reset``
    * [ ] Reinstalled the latest version of [Anaconda](https://www.anaconda.com/download/)
    * [ ] Tried the other applicable steps from the Troubleshooting Guide
* [ ] Completed the **Problem Description**, **Steps to Reproduce** and **Version** sections below


## Problem Description
How to view numbers in a dataframe with thousands separator in spyder variable explorer?


### What steps reproduce the problem?

1. I tried typing %,.2f and %0:,.2f and many other similar combinations in the float formatting filed of the format section but didn't work!
2. 
3. 

### What is the expected output? What do you see instead?



### Paste Traceback/Error Below (if applicable)
<!--- Copy from error dialog or View > Panes > Internal Console --->

```python-traceback

PASTE TRACEBACK HERE

```

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 4.2.0
* Python version: 3.8.5
* Qt version: 
* PyQt version: 
* Operating System name/version: macos 11.1

### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->

```
PASTE DEPENDENCIES HERE
```



Discussion:
Hi @mhdmyz, 

Can you please post a screenshot of what you are seeing after typing `%,.2f and %0:,.2f`?

> Hi @mhdmyz,
> 
> Can you please post a screenshot of what you are seeing after typing `%,.2f and %0:,.2f`?

Hi. Thanks for reply! I get format is incorrect for both of them!
<img width="319" alt="Screen Shot 1399-10-16 at 18 55 09" src="https://user-images.githubusercontent.com/47456155/103664890-01d7d280-4f88-11eb-8215-32d65b26fcd3.png">

@jitseniesen, could you take a look at this one?
I don't think these are valid formats, according to [`printf`-style String Formatting](https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting) in the Python docs.

If we need to provide this functionality, I think we need to use `.format()` instead of the `%` operator to format the numbers.
> I think we need to use .format() instead of the % operator to format the numbers.

That sounds more powerful than our current approach. Do you have time to do that?
No problem, I'll take care of it. It will introduce a minor incompatibility, in that the user will have to input the format string as `.6f` instead of `%.6f`, but config files should remain valid and it will actually simplify the code.

Should I target v4 or v5?
> Should I target v4 or v5?

If the change is small or not disruptive, you can target it for 4.x. Otherwise, it needs to go against master.
did anything ever come of this?  I'd be interested on it too!  thx!
@mg3146 Thanks for the reminder. Nothing seems to have done about this. I don't remember what happened at that time, but it looks like I forgot about this. I will add it to my to-do list.

Resolving patch:
diff --git a/spyder/plugins/variableexplorer/widgets/arrayeditor.py b/spyder/plugins/variableexplorer/widgets/arrayeditor.py
--- a/spyder/plugins/variableexplorer/widgets/arrayeditor.py
+++ b/spyder/plugins/variableexplorer/widgets/arrayeditor.py
@@ -41,48 +41,48 @@
 from spyder.utils.qthelpers import add_actions, create_action, keybinding
 from spyder.plugins.variableexplorer.widgets.basedialog import BaseDialog
 
-# Note: string and unicode data types will be formatted with '%s' (see below)
+# Note: string and unicode data types will be formatted with 's' (see below)
 SUPPORTED_FORMATS = {
-    'single': '%.6g',
-    'double': '%.6g',
-    'float_': '%.6g',
-    'longfloat': '%.6g',
-    'float16': '%.6g',
-    'float32': '%.6g',
-    'float64': '%.6g',
-    'float96': '%.6g',
-    'float128': '%.6g',
-    'csingle': '%r',
-    'complex_': '%r',
-    'clongfloat': '%r',
-    'complex64': '%r',
-    'complex128': '%r',
-    'complex192': '%r',
-    'complex256': '%r',
-    'byte': '%d',
-    'bytes8': '%s',
-    'short': '%d',
-    'intc': '%d',
-    'int_': '%d',
-    'longlong': '%d',
-    'intp': '%d',
-    'int8': '%d',
-    'int16': '%d',
-    'int32': '%d',
-    'int64': '%d',
-    'ubyte': '%d',
-    'ushort': '%d',
-    'uintc': '%d',
-    'uint': '%d',
-    'ulonglong': '%d',
-    'uintp': '%d',
-    'uint8': '%d',
-    'uint16': '%d',
-    'uint32': '%d',
-    'uint64': '%d',
-    'bool_': '%r',
-    'bool8': '%r',
-    'bool': '%r',
+    'single': '.6g',
+    'double': '.6g',
+    'float_': '.6g',
+    'longfloat': '.6g',
+    'float16': '.6g',
+    'float32': '.6g',
+    'float64': '.6g',
+    'float96': '.6g',
+    'float128': '.6g',
+    'csingle': '.6g',
+    'complex_': '.6g',
+    'clongfloat': '.6g',
+    'complex64': '.6g',
+    'complex128': '.6g',
+    'complex192': '.6g',
+    'complex256': '.6g',
+    'byte': 'd',
+    'bytes8': 's',
+    'short': 'd',
+    'intc': 'd',
+    'int_': 'd',
+    'longlong': 'd',
+    'intp': 'd',
+    'int8': 'd',
+    'int16': 'd',
+    'int32': 'd',
+    'int64': 'd',
+    'ubyte': 'd',
+    'ushort': 'd',
+    'uintc': 'd',
+    'uint': 'd',
+    'ulonglong': 'd',
+    'uintp': 'd',
+    'uint8': 'd',
+    'uint16': 'd',
+    'uint32': 'd',
+    'uint64': 'd',
+    'bool_': '',
+    'bool8': '',
+    'bool': '',
 }
 
 
@@ -120,7 +120,7 @@ class ArrayModel(QAbstractTableModel):
     ROWS_TO_LOAD = 500
     COLS_TO_LOAD = 40
 
-    def __init__(self, data, format="%.6g", xlabels=None, ylabels=None,
+    def __init__(self, data, format_spec=".6g", xlabels=None, ylabels=None,
                  readonly=False, parent=None):
         QAbstractTableModel.__init__(self)
 
@@ -145,7 +145,7 @@ def __init__(self, data, format="%.6g", xlabels=None, ylabels=None,
         self.alp = .6 # Alpha-channel
 
         self._data = data
-        self._format = format
+        self._format_spec = format_spec
 
         self.total_rows = self._data.shape[0]
         self.total_cols = self._data.shape[1]
@@ -192,18 +192,18 @@ def __init__(self, data, format="%.6g", xlabels=None, ylabels=None,
             else:
                 self.cols_loaded = self.total_cols
 
-    def get_format(self):
+    def get_format_spec(self):
         """Return current format"""
-        # Avoid accessing the private attribute _format from outside
-        return self._format
+        # Avoid accessing the private attribute _format_spec from outside
+        return self._format_spec
 
     def get_data(self):
         """Return data"""
         return self._data
 
-    def set_format(self, format):
+    def set_format_spec(self, format_spec):
         """Change display format"""
-        self._format = format
+        self._format_spec = format_spec
         self.reset()
 
     def columnCount(self, qindex=QModelIndex()):
@@ -288,7 +288,8 @@ def data(self, index, role=Qt.DisplayRole):
                     return value_to_display(value)
                 else:
                     try:
-                        return to_qvariant(self._format % value)
+                        format_spec = self._format_spec
+                        return to_qvariant(format(value, format_spec))
                     except TypeError:
                         self.readonly = True
                         return repr(value)
@@ -346,7 +347,8 @@ def setData(self, index, value, role=Qt.EditRole):
             return False
 
         # Add change to self.changes
-        self.changes[(i, j)] = val
+        # Use self.test_array to convert to correct dtype
+        self.changes[(i, j)] = self.test_array[0]
         self.dataChanged.emit(index, index)
 
         if not is_string(val):
@@ -559,8 +561,9 @@ def _sel_to_text(self, cell_range):
         _data = self.model().get_data()
         output = io.BytesIO()
         try:
+            fmt = '%' + self.model().get_format_spec()
             np.savetxt(output, _data[row_min:row_max+1, col_min:col_max+1],
-                       delimiter='\t', fmt=self.model().get_format())
+                       delimiter='\t', fmt=fmt)
         except:
             QMessageBox.warning(self, _("Warning"),
                                 _("It was not possible to copy values for "
@@ -592,8 +595,8 @@ def __init__(self, parent, data, readonly=False,
             self.old_data_shape = self.data.shape
             self.data.shape = (1, 1)
 
-        format = SUPPORTED_FORMATS.get(data.dtype.name, '%s')
-        self.model = ArrayModel(self.data, format=format, xlabels=xlabels,
+        format_spec = SUPPORTED_FORMATS.get(data.dtype.name, 's')
+        self.model = ArrayModel(self.data, format_spec=format_spec, xlabels=xlabels,
                                 ylabels=ylabels, readonly=readonly, parent=self)
         self.view = ArrayView(self, self.model, data.dtype, data.shape)
 
@@ -616,18 +619,18 @@ def reject_changes(self):
     @Slot()
     def change_format(self):
         """Change display format"""
-        format, valid = QInputDialog.getText(self, _( 'Format'),
+        format_spec, valid = QInputDialog.getText(self, _( 'Format'),
                                  _( "Float formatting"),
-                                 QLineEdit.Normal, self.model.get_format())
+                                 QLineEdit.Normal, self.model.get_format_spec())
         if valid:
-            format = str(format)
+            format_spec = str(format_spec)
             try:
-                format % 1.1
+                format(1.1, format_spec)
             except:
                 QMessageBox.critical(self, _("Error"),
-                                     _("Format (%s) is incorrect") % format)
+                                     _("Format (%s) is incorrect") % format_spec)
                 return
-            self.model.set_format(format)
+            self.model.set_format_spec(format_spec)
 
 
 class ArrayEditor(BaseDialog):
diff --git a/spyder/plugins/variableexplorer/widgets/dataframeeditor.py b/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
--- a/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
+++ b/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
@@ -41,11 +41,10 @@
 from qtpy.QtCore import (QAbstractTableModel, QModelIndex, Qt, Signal, Slot,
                          QItemSelectionModel, QEvent)
 from qtpy.QtGui import QColor, QCursor
-from qtpy.QtWidgets import (QApplication, QCheckBox, QDialog, QGridLayout,
-                            QHBoxLayout, QInputDialog, QLineEdit, QMenu,
-                            QMessageBox, QPushButton, QTableView,
-                            QScrollBar, QTableWidget, QFrame,
-                            QItemDelegate)
+from qtpy.QtWidgets import (QApplication, QCheckBox, QGridLayout, QHBoxLayout,
+                            QInputDialog, QLineEdit, QMenu, QMessageBox,
+                            QPushButton, QTableView, QScrollBar, QTableWidget,
+                            QFrame, QItemDelegate)
 from spyder_kernels.utils.lazymodules import numpy as np, pandas as pd
 
 # Local imports
@@ -68,7 +67,7 @@
 _bool_false = ['false', 'f', '0', '0.', '0.0', ' ']
 
 # Default format for data frames with floats
-DEFAULT_FORMAT = '%.6g'
+DEFAULT_FORMAT = '.6g'
 
 # Limit at which dataframe is considered so large that it is loaded on demand
 LARGE_SIZE = 5e5
@@ -118,13 +117,13 @@ class DataFrameModel(QAbstractTableModel):
     https://github.com/wavexx/gtabview/blob/master/gtabview/models.py
     """
 
-    def __init__(self, dataFrame, format=DEFAULT_FORMAT, parent=None):
+    def __init__(self, dataFrame, format_spec=DEFAULT_FORMAT, parent=None):
         QAbstractTableModel.__init__(self)
         self.dialog = parent
         self.df = dataFrame
         self.df_columns_list = None
         self.df_index_list = None
-        self._format = format
+        self._format_spec = format_spec
         self.complex_intran = None
         self.display_error_idxs = []
 
@@ -265,14 +264,14 @@ def max_min_col_update(self):
                 max_min = None
             self.max_min_col.append(max_min)
 
-    def get_format(self):
-        """Return current format"""
-        # Avoid accessing the private attribute _format from outside
-        return self._format
+    def get_format_spec(self):
+        """Return current format+spec"""
+        # Avoid accessing the private attribute _format_spec from outside
+        return self._format_spec
 
-    def set_format(self, format):
+    def set_format_spec(self, format_spec):
         """Change display format"""
-        self._format = format
+        self._format_spec = format_spec
         self.reset()
 
     def bgcolor(self, state):
@@ -354,11 +353,11 @@ def data(self, index, role=Qt.DisplayRole):
             value = self.get_value(row, column)
             if isinstance(value, float):
                 try:
-                    return to_qvariant(self._format % value)
+                    return to_qvariant(format(value, self._format_spec))
                 except (ValueError, TypeError):
-                    # may happen if format = '%d' and value = NaN;
+                    # may happen if format = 'd' and value = NaN;
                     # see spyder-ide/spyder#4139.
-                    return to_qvariant(DEFAULT_FORMAT % value)
+                    return to_qvariant(format(value, DEFAULT_FORMAT))
             elif is_type_text_string(value):
                 # Don't perform any conversion on strings
                 # because it leads to differences between
@@ -1017,8 +1016,7 @@ def setup_and_check(self, data, title=''):
         self.setModel(self.dataModel)
         self.resizeColumnsToContents()
 
-        format = '%' + self.get_conf('dataframe_format')
-        self.dataModel.set_format(format)
+        self.dataModel.set_format_spec(self.get_conf('dataframe_format'))
 
         return True
 
@@ -1301,26 +1299,19 @@ def change_format(self):
         """
         Ask user for display format for floats and use it.
         """
-        format, valid = QInputDialog.getText(self, _('Format'),
-                                             _("Float formatting"),
-                                             QLineEdit.Normal,
-                                             self.dataModel.get_format())
+        format_spec, valid = QInputDialog.getText(
+            self, _('Format'), _("Float formatting"), QLineEdit.Normal,
+            self.dataModel.get_format_spec())
         if valid:
-            format = str(format)
+            format_spec = str(format_spec)
             try:
-                format % 1.1
+                format(1.1, format_spec)
             except:
-                msg = _("Format ({}) is incorrect").format(format)
+                msg = _("Format ({}) is incorrect").format(format_spec)
                 QMessageBox.critical(self, _("Error"), msg)
                 return
-            if not format.startswith('%'):
-                msg = _("Format ({}) should start with '%'").format(format)
-                QMessageBox.critical(self, _("Error"), msg)
-                return
-            self.dataModel.set_format(format)
-
-            format = format[1:]
-            self.set_conf('dataframe_format', format)
+            self.dataModel.set_format_spec(format_spec)
+            self.set_conf('dataframe_format', format_spec)
 
     def get_value(self):
         """Return modified Dataframe -- this is *not* a copy"""


Test patch:
diff --git a/spyder/plugins/variableexplorer/widgets/tests/test_arrayeditor.py b/spyder/plugins/variableexplorer/widgets/tests/test_arrayeditor.py
--- a/spyder/plugins/variableexplorer/widgets/tests/test_arrayeditor.py
+++ b/spyder/plugins/variableexplorer/widgets/tests/test_arrayeditor.py
@@ -127,14 +127,25 @@ def test_arrayeditor_format(setup_arrayeditor, qtbot):
     qtbot.keyClick(dlg.arraywidget.view, Qt.Key_Down, modifier=Qt.ShiftModifier)
     contents = dlg.arraywidget.view._sel_to_text(dlg.arraywidget.view.selectedIndexes())
     assert contents == "1\n2\n"
-    dlg.arraywidget.view.model().set_format("%.18e")
-    assert dlg.arraywidget.view.model().get_format() == "%.18e"
+    dlg.arraywidget.view.model().set_format_spec(".18e")
+    assert dlg.arraywidget.view.model().get_format_spec() == ".18e"
     qtbot.keyClick(dlg.arraywidget.view, Qt.Key_Down, modifier=Qt.ShiftModifier)
     qtbot.keyClick(dlg.arraywidget.view, Qt.Key_Down, modifier=Qt.ShiftModifier)
     contents = dlg.arraywidget.view._sel_to_text(dlg.arraywidget.view.selectedIndexes())
     assert contents == "1.000000000000000000e+00\n2.000000000000000000e+00\n"
 
 
+@pytest.mark.parametrize(
+    'data',
+    [np.array([10000])]
+)
+def test_arrayeditor_format_thousands(setup_arrayeditor):
+    """Check that format can include thousands separator."""
+    model = setup_arrayeditor.arraywidget.model
+    model.set_format_spec(',.2f')
+    assert model.data(model.index(0, 0)) == '10,000.00'
+
+
 def test_arrayeditor_with_inf_array(qtbot, recwarn):
     """See: spyder-ide/spyder#8093"""
     arr = np.array([np.inf])
diff --git a/spyder/plugins/variableexplorer/widgets/tests/test_dataframeeditor.py b/spyder/plugins/variableexplorer/widgets/tests/test_dataframeeditor.py
--- a/spyder/plugins/variableexplorer/widgets/tests/test_dataframeeditor.py
+++ b/spyder/plugins/variableexplorer/widgets/tests/test_dataframeeditor.py
@@ -348,40 +348,39 @@ def test_dataframemodel_get_bgcolor_with_missings():
 
 def test_dataframemodel_with_format_percent_d_and_nan():
     """
-    Test DataFrameModel with format `%d` and dataframe containing NaN
+    Test DataFrameModel with format `d` and dataframe containing NaN
 
     Regression test for spyder-ide/spyder#4139.
     """
     np_array = numpy.zeros(2)
     np_array[1] = numpy.nan
     dataframe = DataFrame(np_array)
-    dfm = DataFrameModel(dataframe, format='%d')
+    dfm = DataFrameModel(dataframe, format_spec='d')
     assert data(dfm, 0, 0) == '0'
     assert data(dfm, 1, 0) == 'nan'
 
 def test_change_format(qtbot, monkeypatch):
     mockQInputDialog = Mock()
-    mockQInputDialog.getText = lambda parent, title, label, mode, text: ('%10.3e', True)
+    mockQInputDialog.getText = lambda parent, title, label, mode, text: ('10.3e', True)
     monkeypatch.setattr('spyder.plugins.variableexplorer.widgets.dataframeeditor.QInputDialog', mockQInputDialog)
     df = DataFrame([[0]])
     editor = DataFrameEditor(None)
     editor.setup_and_check(df)
     editor.change_format()
-    assert editor.dataModel._format == '%10.3e'
+    assert editor.dataModel._format_spec == '10.3e'
     assert editor.get_conf('dataframe_format') == '10.3e'
     editor.set_conf('dataframe_format', '.6g')
 
-def test_change_format_with_format_not_starting_with_percent(qtbot, monkeypatch):
-    mockQInputDialog = Mock()
-    mockQInputDialog.getText = lambda parent, title, label, mode, text: ('xxx%f', True)
-    monkeypatch.setattr('spyder.plugins.variableexplorer.widgets.dataframeeditor'
-                        '.QInputDialog', mockQInputDialog)
-    monkeypatch.setattr('spyder.plugins.variableexplorer.widgets.dataframeeditor'
-                        '.QMessageBox.critical', Mock())
-    df = DataFrame([[0]])
-    editor = DataFrameEditor(None)
-    editor.setup_and_check(df)
-    editor.change_format()
+
+def test_dataframemodel_with_format_thousands():
+    """
+    Check that format can include thousands separator.
+
+    Regression test for spyder-ide/spyder#14518.
+    """
+    dataframe = DataFrame([10000.1])
+    dfm = DataFrameModel(dataframe, format_spec=',.2f')
+    assert data(dfm, 0, 0) == '10,000.10'
 
 
 def test_dataframeeditor_with_various_indexes():
diff --git a/spyder/widgets/tests/test_collectioneditor.py b/spyder/widgets/tests/test_collectioneditor.py
--- a/spyder/widgets/tests/test_collectioneditor.py
+++ b/spyder/widgets/tests/test_collectioneditor.py
@@ -273,7 +273,7 @@ def test_create_dataframeeditor_with_correct_format(qtbot):
     editor.delegate.createEditor(None, None, editor.model.index(0, 3))
     dataframe_editor = next(iter(editor.delegate._editors.values()))['editor']
     qtbot.addWidget(dataframe_editor)
-    dataframe_editor.dataModel._format == '%10d'
+    dataframe_editor.dataModel._format_spec == '10d'
 
 
 def test_collectionsmodel_with_two_ints():