internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: vaexio/vaex

Issue:
Columns starting with a number generate Errors
The latest version of Vaex cannot read files when columns are starting with a number.

This was not the case before. I understand the issue of such names when doing expressions etc, but the reading of the data was working before.

```python
/usr/local/lib/python3.6/dist-packages/vaex/hdf5/dataset.py in _load_columns(self, h5data, first)
    356         self.column_names = []
    357         for name in column_order:
--> 358             self.columns[name] = all_columns.pop(name)
    359             self.column_names.append(name)
    360         # add the rest

KeyError: '2MASS_J'
```


Discussion:
Hi Morgan,

thanks for the report, Is this a file you generated yourself, or one you generated with vaex?

cheers,

Maarten
Hi Maarten,
With vaex but a while ago. It was working well until recently.

Resolving patch:
diff --git a/packages/vaex-hdf5/vaex/hdf5/dataset.py b/packages/vaex-hdf5/vaex/hdf5/dataset.py
--- a/packages/vaex-hdf5/vaex/hdf5/dataset.py
+++ b/packages/vaex-hdf5/vaex/hdf5/dataset.py
@@ -352,7 +352,8 @@ def check_format(self, *args, **kwargs):
                             transposed = shape[1] < shape[0]
                             self.addRank1(column_name, offset, shape[1], length1=shape[0], dtype=data.dtype, stride=1, stride1=1, transposed=transposed)
         all_columns = dict(**self.columns)
-        # print(all_columns, column_order)
+        # in case the column_order refers to non-existing columns
+        column_order = [k for k in column_order if k in all_columns]
         self.column_names = []
         for name in column_order:
             self.columns[name] = all_columns.pop(name)


Test patch:
diff --git a/tests/column_names_test.py b/tests/column_names_test.py
--- a/tests/column_names_test.py
+++ b/tests/column_names_test.py
@@ -1,6 +1,6 @@
 import vaex
 from common import *
-
+import h5py
 
 x = np.arange(10)
 
@@ -35,3 +35,19 @@ def test_add_invalid_name(tmpdir):
     df = vaex.open(path)
     assert df['X!1'].tolist() == x.tolist()
     assert (df.copy()['X!1']*2).tolist() == (x*2).tolist()
+
+def test_invalid_name_read(tmpdir):
+    # earlier version of vaex could write invalid names, check if we can read those
+    df = vaex.from_dict({'x': x})
+    # df.columns['1'] = df.columns.pop('x')
+    # df.column_names = ['1']
+    path = str(tmpdir.join('test.hdf5'))
+    df.export(path)
+
+    h5 = h5py.File(path)
+    h5['/table/columns']['1'] = h5['/table/columns']['x']
+    del h5['/table/columns']['x']
+
+    df = vaex.open(path)
+    assert df['1'].tolist() == x.tolist()
+    assert (df.copy()['1']*2).tolist() == (x*2).tolist()
Repository: vaexio/vaex

Issue:
dtype automatic conversion when needed
There is some inconsistent behaviour between `numpy` and `vaex` when it comes to operations on floats and ints that have lower bits than the result. Consider the following example:

```
import vaex
import numpy as np

x = np.ones(500, dtype=np.int8)
df = vaex.from_arrays(x=x)

print('Numpy:', x.sum())
print('Vaex:', df.x.sum())
print('Vaex (manual conversion):', df.x.astype('int').sum())
```
For the operation `x.sum()`, `numpy` uses the default `int64` type even though `x` is of type `int8`, and will return a `int64`. On the other hand, `vaex` tries to do the same, however the output of operations such as `sum` or `mean` return an `array`  with the same `dtype` as the column/expression in which this method was applied. In the above example, the result of the sum (500) will be converted to an array of type `int8`, hence the value -12.

The question is how do we want to handle this case. Should all output arrays be cast to the python default `int64` or `float64`, or do we want to keep this behaviour?


Discussion:

Resolving patch:
diff --git a/packages/vaex-core/vaex/agg.py b/packages/vaex-core/vaex/agg.py
--- a/packages/vaex-core/vaex/agg.py
+++ b/packages/vaex-core/vaex/agg.py
@@ -62,6 +62,8 @@ def _create_operation(self, df, grid):
             self.dtype_out = self.dtype_in
             if self.short_name == "count":
                 self.dtype_out = np.dtype('int64')
+            if self.short_name in ['sum', 'summoment']:
+                self.dtype_out = vaex.utils.upcast(self.dtype_in)
         agg_op_type = vaex.utils.find_type_from_dtype(vaex.superagg, self.name + "_", self.dtype_in)
         agg_op = agg_op_type(grid, *self.agg_args)
         return agg_op
diff --git a/packages/vaex-core/vaex/utils.py b/packages/vaex-core/vaex/utils.py
--- a/packages/vaex-core/vaex/utils.py
+++ b/packages/vaex-core/vaex/utils.py
@@ -915,3 +915,15 @@ def unmask_selection_mask(selection_mask):
         # exclude the masked values
         selection_mask = selection_mask & ~mask
     return selection_mask
+
+
+def upcast(dtype):
+    if dtype.kind == "b":
+        return np.dtype('int64')
+    if dtype.kind == "i":
+        return np.dtype('int64')
+    if dtype.kind == "u":
+        return np.dtype('uint64')
+    if dtype.kind == "f":
+        return np.dtype('float64')
+    return dtype


Test patch:
diff --git a/tests/agg_test.py b/tests/agg_test.py
--- a/tests/agg_test.py
+++ b/tests/agg_test.py
@@ -205,3 +205,12 @@ def test_agg_selections():
     assert df_grouped['z_sum_selected'].tolist() == [2, 4, 13]
     assert df_grouped['z_mean_selected'].tolist() == [1, 4, 6.5]
     assert df_grouped['w_nuniqe_selected'].tolist() == [2, 1, 2]
+
+def test_upcast():
+    df = vaex.from_arrays(b=[False, True, True], i8=np.array([120, 121, 122], dtype=np.int8),
+        f4=np.array([1, 1e-13, 1], dtype=np.float32))
+    assert df.b.sum() == 2
+    assert df.i8.sum() == 120*3 + 3
+    assert df.f4.sum() == (2 + 1e-13)
+
+    assert abs(df.b.var() - (0.2222)) < 0.01
Repository: vaexio/vaex

Issue:
Support sparse data
The data in many domains, such as scRNA, is sparse. 


Discussion:
We have added some experimetal sparse support (csr_matrix), but I'd rather wait for Apache Arrow #517 to look more into this.
In case you want to try:
```python
from scipy.sparse import csc_matrix, csr_matrix

def test_sparse_basics():
    x = np.arange(3)
    s = csr_matrix([[0, 1], [1, 0], [2, 3]])
    ds = vaex.from_arrays(x=x)
    ds.add_columns(['s1', 's2'], s)
    assert ds.s1.tolist() == [0, 1, 2]
    assert ds.s2.tolist() == [1, 0, 3]

```
Note that this example fails if you try to print ds

Resolving patch:
diff --git a/packages/vaex-core/vaex/column.py b/packages/vaex-core/vaex/column.py
--- a/packages/vaex-core/vaex/column.py
+++ b/packages/vaex-core/vaex/column.py
@@ -36,7 +36,7 @@ def trim(self, i1, i2):
         return ColumnVirtualRange(self.start + i1 * self.step, self.start + i2 * self.step, self.step, self.dtype)
 
 
-class ColumnSparse(object):
+class ColumnSparse(Column):
     def __init__(self, matrix, column_index):
         self.matrix = matrix
         self.column_index = column_index
diff --git a/packages/vaex-core/vaex/dataframe.py b/packages/vaex-core/vaex/dataframe.py
--- a/packages/vaex-core/vaex/dataframe.py
+++ b/packages/vaex-core/vaex/dataframe.py
@@ -4954,8 +4954,9 @@ def copy(self, column_names=None, virtual=True):
                 added.add(name)
                 if name in self.columns:
                     column = self.columns[name]
-                    if not isinstance(column, ColumnSparse):
-                        df.add_column(name, column, dtype=self._dtypes_override.get(name))
+                    df.add_column(name, column, dtype=self._dtypes_override.get(name))
+                    if isinstance(column, ColumnSparse):
+                        df._sparse_matrices[name] = self._sparse_matrices[name]
                 elif name in self.virtual_columns:
                     if virtual:  # TODO: check if the ast is cached
                         df.add_virtual_column(name, self.virtual_columns[name])


Test patch:
diff --git a/tests/sparse_test.py b/tests/sparse_test.py
new file mode 100644
--- /dev/null
+++ b/tests/sparse_test.py
@@ -0,0 +1,29 @@
+from common import *
+from scipy.sparse import csc_matrix, csr_matrix
+
+
+def test_sparse_basics():
+    x = np.arange(3)
+    s = csr_matrix([[0, 1], [1, 0], [2, 3]])
+    ds = vaex.from_arrays(x=x)
+    ds.add_columns(['s1', 's2'], s)
+    assert ds.s1.tolist() == [0, 1, 2]
+    assert ds.s2.tolist() == [1, 0, 3]
+    assert "error" not in repr(ds)
+
+
+def test_sparse_export(tmpdir):
+    path = str(tmpdir.join('test.hdf5'))
+    x = np.arange(3)
+    ds = vaex.from_arrays(x=x)
+    s = csr_matrix([[0, 1], [1, 0], [2, 3]])
+    ds.add_columns(['s1', 's2'], s)
+
+    p = csr_matrix([[4, 0], [0, 0], [9, 10]])
+    ds.add_columns(['p1', 'p2'], p)
+
+    ds.export_hdf5(path)
+
+    ds2 = vaex.open(path)
+    assert ds2.s1.tolist() == [0, 1, 2]
+    assert ds2.s2.tolist() == [1, 0, 3]
Repository: vaexio/vaex

Issue:
Error printing DataFrames with certain types of virtual columns
When trying to print or display a dataframe with a certain type of virtual columngs, but only using a subset of the columns crashes with `NameError: name 'do_not_understand_expression' is not defined`.

See a fully reproducible example below.

```
import vaex

df = vaex.from_arrays(x=['dog', 'cat', 'cat', 'dog', 'mouse'], time=[2230, 0, 1200, 1515, 945])
# Extract the hour from the time column
df['hour'] = df.time // 100 % 24

# Printing/displaying the  entire DataFrame works:
print(df)

# Printing/displaying a subset of columns, which includes the virtual column created above crashes
print(df[['x', 'hour']])
```


Discussion:

Resolving patch:
diff --git a/packages/vaex-core/vaex/expresso.py b/packages/vaex-core/vaex/expresso.py
--- a/packages/vaex-core/vaex/expresso.py
+++ b/packages/vaex-core/vaex/expresso.py
@@ -357,6 +357,8 @@ def visit_BinOp(self, node):
                 return "({left} * {right})".format(left=left, right=right)
             if isinstance(node.op, ast.Div):
                 return "({left} / {right})".format(left=left, right=right)
+            if isinstance(node.op, ast.FloorDiv):
+                return "({left} // {right})".format(left=left, right=right)
             if isinstance(node.op, ast.Add):
                 return "({left} + {right})".format(left=left, right=right)
             if isinstance(node.op, ast.Sub):


Test patch:
diff --git a/tests/expresso_test.py b/tests/expresso_test.py
new file mode 100644
--- /dev/null
+++ b/tests/expresso_test.py
@@ -0,0 +1,36 @@
+from vaex.expresso import parse_expression, node_to_string, simplify, translate
+
+
+def test_compare():
+    expr = "(x < 0)"
+    node = parse_expression(expr)
+    assert expr == node_to_string(node)
+
+    expr = "((x < 0) >= 6)"
+    node = parse_expression(expr)
+    assert expr == node_to_string(node)
+
+    expr = "(x // 10)"
+    node = parse_expression(expr)
+    assert expr == node_to_string(node)
+
+def test_simplify():
+    assert simplify("0 + 1") == "1"
+    assert simplify("1 + 0") == "1"
+    assert simplify("-0 + 1") == "1"
+    assert simplify("1 + -0") == "1"
+
+    assert simplify("0 * a + b") == "b"
+    assert simplify("a * 0 + b") == "b"
+    assert simplify("b + 0 * a") == "b"
+    assert simplify("b + a * 0") == "b"
+
+
+def test_lists():
+    node = parse_expression("searchsorted(['9E', 'AA', 'AQ', 'AS', 'B6', 'CO', 'DL', 'EV', 'F9', 'FL', 'HA', 'MQ', 'NW', 'OH', 'OO', 'UA', 'US', 'WN', 'XE', 'YV'], UniqueCarrier)")
+    assert node is not None
+
+    expr = "searchsorted(['9E', 'AA', 'AQ', 'AS', 'B6', 'CO', 'DL', 'EV', 'F9', 'FL', 'HA', 'MQ', 'NW', 'OH', 'OO', 'UA', 'US', 'WN', 'XE', 'YV'], UniqueCarrier)"
+    expr_translate = translate(expr, lambda x: None)
+    print(node)
+    assert expr == expr_translate
Repository: vaexio/vaex

Issue:
Bool values get flipped when converting Arrow table to DataFrame
Using the latest version:
`vaex==2.6.1`

Just realised that when converting an Arrow table to a DataFrame, bool columns get flipped and converted to integers:

```python
import vaex
from pyarrow import feather

bool_array = [False, True, True, False]
pdf = pd.DataFrame({"col1": bool_array})
pdf.to_feather("test_data.feather")
arrow_table = feather.read_table("test_data.feather")
vaex.from_arrow_table(arrow_table)
```

```
#  | col1
-- | --
0  | 1
1  | 0
2  | 0
3  | 1
```


Discussion:
Thanks for the report, and the reproducible example!


Resolving patch:
diff --git a/packages/vaex-arrow/vaex_arrow/convert.py b/packages/vaex-arrow/vaex_arrow/convert.py
--- a/packages/vaex-arrow/vaex_arrow/convert.py
+++ b/packages/vaex-arrow/vaex_arrow/convert.py
@@ -53,6 +53,7 @@ def numpy_array_from_arrow_array(arrow_array):
     buffers = arrow_array.buffers()
     assert len(buffers) == 2
     bitmap_buffer, data_buffer = buffers
+    offset = arrow_array.offset
     if isinstance(arrow_type, type(pyarrow.binary(1))):  # todo, is there a better way to typecheck?
         # mimics python/pyarrow/array.pxi::Array::to_numpy
         assert len(buffers) == 2
@@ -68,13 +69,13 @@ def numpy_array_from_arrow_array(arrow_array):
     if np.bool_ == dtype:
         # TODO: this will also be a copy, we probably want to support bitmasks as well
         bitmap = np.frombuffer(data_buffer, np.uint8, len(data_buffer))
-        array = numpy_mask_from_arrow_mask(bitmap, len(arrow_array))
+        array = numpy_bool_from_arrow_bitmap(bitmap, len(arrow_array) + offset)[offset:]
     else:
-        array = np.frombuffer(data_buffer, dtype, len(arrow_array))
+        array = np.frombuffer(data_buffer, dtype, len(arrow_array) + offset)[offset:]
 
     if bitmap_buffer is not None:
         bitmap = np.frombuffer(bitmap_buffer, np.uint8, len(bitmap_buffer))
-        mask = numpy_mask_from_arrow_mask(bitmap, len(arrow_array))
+        mask = numpy_mask_from_arrow_mask(bitmap, len(arrow_array) + offset)[offset:]
         array = np.ma.MaskedArray(array, mask=mask)
     return array
 
@@ -83,7 +84,10 @@ def numpy_mask_from_arrow_mask(bitmap, length):
     # we do have to change the ordering of the bits
     return 1-np.unpackbits(bitmap).reshape((len(bitmap),8))[:,::-1].reshape(-1)[:length]
 
-
+def numpy_bool_from_arrow_bitmap(bitmap, length):
+    # arrow uses a bitmap https://github.com/apache/arrow/blob/master/format/Layout.md
+    # we do have to change the ordering of the bits
+    return np.unpackbits(bitmap).reshape((len(bitmap),8))[:,::-1].reshape(-1)[:length].view(np.bool_)
 
 def arrow_table_from_vaex_df(ds, column_names=None, selection=None, strings=True, virtual=False):
     """Implementation of Dataset.to_arrow_table"""


Test patch:
Repository: vaexio/vaex

Issue:
[BUG-REPORT] TypeError: can't pickle vaex.superutils.ordered_set
**Description**
If I use `df.func.where` with `isin`, I can't pickle the resulted state.
This is for machine learning pipelines. 

reproduce:
```
import vaex
import pickle

df = vaex.from_dict({'a':[1,2,3]})
df['b'] = df.func.where(df['a'].isin([1]),1,2)
pickle.dumps(df.state_get())
...
TypeError: can't pickle vaex.superutils.ordered_set_int64 objects
```


**Software information**
 - vaex-core==4.0.0a11
 - Vaex was installed via: pip
 - OS: Mac
 


Discussion:

Resolving patch:
diff --git a/packages/vaex-core/vaex/hash.py b/packages/vaex-core/vaex/hash.py
--- a/packages/vaex-core/vaex/hash.py
+++ b/packages/vaex-core/vaex/hash.py
@@ -1,12 +1,19 @@
 import os
+import copyreg
 
 
 on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
 if not on_rtd:
     from .superutils import *
     from . import superutils
+
     ordered_set = tuple([cls for name, cls in vars(superutils).items() if name.startswith('ordered_set')])
 
+    def pickle(x):
+        return type(x), (x.extract(), x.count, x.nan_count, x.null_count)
+    for cls in ordered_set:
+        copyreg.pickle(cls, pickle)
+
 
 def counter_type_from_dtype(dtype, transient=True):
     from .array_types import is_string_type
@@ -24,6 +31,7 @@ def counter_type_from_dtype(dtype, transient=True):
     name = 'counter_' + postfix
     return globals()[name]
 
+
 def ordered_set_type_from_dtype(dtype, transient=True):
     from .array_types import is_string_type
     if is_string_type(dtype):
@@ -38,6 +46,7 @@ def ordered_set_type_from_dtype(dtype, transient=True):
     name = 'ordered_set_' + postfix
     return globals()[name]
 
+
 def index_type_from_dtype(dtype, transient=True, prime_growth=False):
     from .array_types import is_string_type
     if is_string_type(dtype):
@@ -53,7 +62,3 @@ def index_type_from_dtype(dtype, transient=True, prime_growth=False):
     if prime_growth:
         name += "_prime_growth"
     return globals()[name]
-
-# from numpy import *
-# import IPython
-# IPython.embed()
\ No newline at end of file


Test patch:
diff --git a/tests/pickle_test.py b/tests/pickle_test.py
--- a/tests/pickle_test.py
+++ b/tests/pickle_test.py
@@ -121,3 +121,12 @@ def test_concat(df_file, tmpdir):
     assert df2.x.count() == len(df_file) * 2, 'x is repeated'
     assert df2.x.sum() == df_file.x.sum() * 2, 'x is repeated'
     assert df2.y.sum() == df_file.y.sum(), 'y is not repeated'
+
+
+def test_state_with_set():
+    df = vaex.from_arrays(x=[1,2,3])
+    df['test'] = df.x.isin([1,2])
+    df2 = pickle.loads(pickle.dumps(df))
+    assert df2.x.tolist() == df.x.tolist()
+    # make sure the state itself can be pickled, not just the dataframe
+    pickle.dumps(df.state_get())
Repository: vaexio/vaex

Issue:
[BUG-REPORT] Can no longer pass glob options to vaex.open
**Description**
There seems to be a problem with columns that have spaces in them. I provided an example below that clearly demonstrates it.

````

bx = vaex.open(
    f".tmp/short_data/NQBXshvol*.csv",
    convert=True,
    sep='|',
    copy_index=False,
    skip_blank_lines=True,
)

ERROR:MainThread:vaex:error opening '.tmp/short_data/NQBXshvol*.csv'
generating bx
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-15-9be15d450091> in <module>
     45 else:
     46     print('generating bx')
---> 47     bx = vaex.open(
     48         f".tmp/short_data/NQBXshvol*.csv",
     49         convert=True,

~/.pyenv/versions/3.9.1/lib/python3.9/site-packages/vaex/__init__.py in open(path, convert, shuffle, fs_options, fs, *args, **kwargs)
    191                 naked_path, options = vaex.file.split_options(path)
--> 193                 if glob.has_magic(naked_path):
    194                     filenames.extend(list(sorted(vaex.file.glob(path, **kwargs))))
    195                 else:

TypeError: glob() got an unexpected keyword argument 'sep'

````


Discussion:
I ran into the same problem

Resolving patch:
diff --git a/packages/vaex-core/vaex/__init__.py b/packages/vaex-core/vaex/__init__.py
--- a/packages/vaex-core/vaex/__init__.py
+++ b/packages/vaex-core/vaex/__init__.py
@@ -196,7 +196,7 @@ def open(path, convert=False, shuffle=False, fs_options={}, fs=None, *args, **kw
             path = vaex.file.stringyfy(path)
             naked_path, options = vaex.file.split_options(path)
             if glob.has_magic(naked_path):
-                filenames.extend(list(sorted(vaex.file.glob(path, **kwargs))))
+                filenames.extend(list(sorted(vaex.file.glob(path, fs_options=fs_options, fs=fs))))
             else:
                 filenames.append(path)
         df = None


Test patch:
Repository: vaexio/vaex

Issue:
The html tag escapes for Vaex index column in jupyterlab 
I install vaex 2.6.1 in jupyterlab 1.2.7. When I print vaex dataframe, the index column # show html tag.
![image](https://user-images.githubusercontent.com/34529711/79339972-45069700-7f5c-11ea-8840-d8779f587902.png)

I inspect the index element. It looks like the "<" and ">" been encode to "&lt;" and "&gt;"
![image](https://user-images.githubusercontent.com/34529711/79340322-b5151d00-7f5c-11ea-9845-3a38370919d5.png)

Can you help to check it or avoid htmlEscapes for index column #?



Discussion:
Appearently tabulate 0.8.7 broke html escaping https://github.com/astanin/python-tabulate/issues/48
I suggest you downgrade it to 0.8.6, and once 0.8.8 is out we can apply a workaround.
Version 0.8.9 is now out. Would it be possible to fix it?

Resolving patch:
diff --git a/packages/vaex-core/vaex/dataframe.py b/packages/vaex-core/vaex/dataframe.py
--- a/packages/vaex-core/vaex/dataframe.py
+++ b/packages/vaex-core/vaex/dataframe.py
@@ -3607,10 +3607,9 @@ def table_part(k1, k2, parts):
         # print(values_list)
         import tabulate
         table_text = str(tabulate.tabulate(values_list, headers="keys", tablefmt=format))
-        if tabulate.__version__ == '0.8.7':
-            # Tabulate 0.8.7 escapes html :()
-            table_text = table_text.replace('&lt;i style=&#x27;opacity: 0.6&#x27;&gt;', "<i style='opacity: 0.6'>")
-            table_text = table_text.replace('&lt;/i&gt;', "</i>")
+        # Tabulate 0.8.7+ escapes html :()
+        table_text = table_text.replace('&lt;i style=&#x27;opacity: 0.6&#x27;&gt;', "<i style='opacity: 0.6'>")
+        table_text = table_text.replace('&lt;/i&gt;', "</i>")
         if i2 - i1 == 0:
             if self._length_unfiltered != len(self):
                 footer_text = 'No rows to display (because of filtering).'


Test patch:
Repository: vaexio/vaex

Issue:
Displaying full integers instead of scientific notiation when printing out Vaex HDF5 data
Hi,
Can you help me:
my code:
`myfile = vaex.open('myfile.hdf5')`
`myfile['customer_id']`

output:
`Length: 4,259,376 dtype: int64 (column)`
`0   9.4618e+08`
` 1  9.43324e+08`
`  2  9.43325e+08`
`  3  9.43333e+08`
` 4  9.43333e+08`
   ...

How can get output with full integer instead of scientific notation? Thank you.



Discussion:

Resolving patch:
diff --git a/packages/vaex-core/vaex/formatting.py b/packages/vaex-core/vaex/formatting.py
--- a/packages/vaex-core/vaex/formatting.py
+++ b/packages/vaex-core/vaex/formatting.py
@@ -8,25 +8,32 @@
 MAX_LENGTH = 50
 
 
+def _trim_string(value):
+    if len(value) > MAX_LENGTH:
+        value = repr(value[:MAX_LENGTH-3])[:-1] + '...'
+    return value
+
 def _format_value(value):
-    if isinstance(value, six.string_types):
-        value = str(value)
-    elif isinstance(value, pa.lib.Scalar):
+    # print("value = ", value, type(value), isinstance(value, numbers.Number))
+    if isinstance(value, pa.lib.Scalar):
         value = value.as_py()
         if value is None:
-            value = '--'
+            return '--'
         else:
-            value = repr(value)
+            return _trim_string(str(value))
+    if isinstance(value, str):
+        return _trim_string(str(value))
     elif isinstance(value, bytes):
-        value = repr(value)
+        value = _trim_string(repr(value))
     elif isinstance(value, np.ma.core.MaskedConstant):
-        value = str(value)
-    if isinstance(value, np.datetime64):
+        return str(value)
+    elif isinstance(value, np.datetime64):
         if np.isnat(value):
             value = 'NaT'
         else:
             value = ' '.join(str(value).split('T'))
-    if isinstance(value, np.timedelta64):
+        return value
+    elif isinstance(value, np.timedelta64):
         if np.isnat(value):
             value = 'NaT'
         else:
@@ -40,11 +47,10 @@ def _format_value(value):
                 value = str('%i days %+02i:%02i:%02i.%i' % (d,h,m,s,ms))
             else:
                 value = str('%i days %+02i:%02i:%02i' % (d,h,m,s))
-    elif not isinstance(value, numbers.Number):
+        return value
+    elif isinstance(value, numbers.Number):
         value = str(value)
-    if isinstance(value, float):
+    else:
         value = repr(value)
-    if isinstance(value, (str, bytes)):
-        if len(value) > MAX_LENGTH:
-            value = repr(value[:MAX_LENGTH-3])[:-1] + '...'
+        value = _trim_string(value)
     return value


Test patch:
diff --git a/tests/repr_test.py b/tests/repr_test.py
--- a/tests/repr_test.py
+++ b/tests/repr_test.py
@@ -67,3 +67,10 @@ def test_slice_filtered_remte(ds_remote):
     df = ds_remote
     dff = df[df.x > 0]
     assert "0.0bla" not in repr(dff[['x']])
+
+
+def test_display_large_int(df_factory):
+    large_int = 2**50-1
+    df = df_factory(x=[123, large_int])
+    text = repr(df)
+    assert str(large_int) in text
\ No newline at end of file