Repository: vaexio/vaex

Issue:
[BUG-REPORT] cannot process multibyte column names with vaex.BinnerTime
**Description**
Suppose I have a vaex dataframe `df` read from pandas as below. It contains a datetime column named in Chinese (字段1, literally 'field 1') and a string column (字段2).

\#  | 字段1 | 字段2
--|-----:|:---
0 | 2009-1-1 | 一
1 | 2009-2-1 | 二

Some testcases:

- Passed tests
  - `df.groupby('字段2').agg('count')` 
  - `df.groupby(df.字段2).agg('count')`
- Failed tests
  - `df.groupby(vaex.BinnerTime(df.字段1)).agg('count')`
  - `df.groupby(vaex.BinnerTime(u'字段1')).agg('count')`

Error message:

```python
ValueError                                Traceback (most recent call last)
<ipython-input-59-b91ac46b6ba4> in <module>
----> 1 df.groupby(vaex.BinnerTime(df.字段1)).agg('count')

C:\ProgramData\anaconda3\lib\site-packages\vaex\groupby.py in __init__(self, expression, resolution, df, every)
     57         self.begin_name = self.df.add_variable('t_begin', self.tmin.astype(self.resolution_type), unique=True)
     58         # TODO: import integer from future?
---> 59         self.binby_expression = str(self.df['%s - %s' % (self.expression.astype(self.resolution_type), self.begin_name)].astype('int') // every)
     60         self.binner = self.df._binner_ordinal(self.binby_expression, self.N)
     61 

C:\ProgramData\anaconda3\lib\site-packages\vaex\dataframe.py in __getitem__(self, item)
   4585             # if item in self._virtual_expressions:
   4586             #     return self._virtual_expressions[item]
-> 4587             self.validate_expression(item)
   4588             return Expression(self, item)  # TODO we'd like to return the same expression if possible
   4589         elif isinstance(item, Expression):

C:\ProgramData\anaconda3\lib\site-packages\vaex\dataframe.py in validate_expression(self, expression)
   2899         funcs = set(expression_namespace.keys())  | set(self.functions.keys())
   2900         try:
-> 2901             return vaex.expresso.validate_expression(expression, vars, funcs)
   2902         except NameError as e:
   2903             raise NameError(str(e)) from None

C:\ProgramData\anaconda3\lib\site-packages\vaex\expresso.py in validate_expression(expr, variable_set, function_set, names)
     77                              type(node.body))
     78         validate_expression(first_expr.value, variable_set,
---> 79                             function_set, names)
     80     elif isinstance(expr, _ast.BinOp):
     81         if expr.op.__class__ in valid_binary_operators:

C:\ProgramData\anaconda3\lib\site-packages\vaex\expresso.py in validate_expression(expr, variable_set, function_set, names)
     81         if expr.op.__class__ in valid_binary_operators:
     82             validate_expression(expr.right, variable_set, function_set, names)
---> 83             validate_expression(expr.left, variable_set, function_set, names)
     84         else:
     85             raise ValueError("Binary operator not allowed: %r" % expr.op)

C:\ProgramData\anaconda3\lib\site-packages\vaex\expresso.py in validate_expression(expr, variable_set, function_set, names)
    108         last_func = expr
    109         for arg in expr.args:
--> 110             validate_expression(arg, variable_set, function_set, names)
    111         for arg in expr.keywords:
    112             validate_expression(arg, variable_set, function_set, names)

C:\ProgramData\anaconda3\lib\site-packages\vaex\expresso.py in validate_expression(expr, variable_set, function_set, names)
     91             raise ValueError("Unary operator not allowed: %r" % expr.op)
     92     elif isinstance(expr, _ast.Name):
---> 93         validate_id(expr.id)
     94         if expr.id not in variable_set:
     95             matches = difflib.get_close_matches(expr.id, list(variable_set))

C:\ProgramData\anaconda3\lib\site-packages\vaex\expresso.py in validate_id(id)
    570     for char in id:
    571         if char not in valid_id_characters:
--> 572             raise ValueError("invalid character %r in id %r" % (char, id))

ValueError: invalid character '字' in id '字段1'
```

**Software information**
 - Vaex version (`import vaex; vaex.__version__)`:
{'vaex': '3.0.0',
 'vaex-core': '2.0.3',
 'vaex-viz': '0.4.0',
 'vaex-hdf5': '0.6.0',
 'vaex-server': '0.3.1',
 'vaex-astro': '0.7.0',
 'vaex-jupyter': '0.5.2',
 'vaex-ml': '0.9.0',
 'vaex-arrow': '0.5.1'}
 - Vaex was installed via: pip
 - OS: Windows Server 2019 Datacenter (build 1809)

**Additional information**




Discussion:
Hm.. I just tested this the other day and it seemed to work fine on my end. 

Can you please try (as a test only) to rename some of the columns, and see if the operation will work on columns that are named using latin-only characters? It looks to me the issue is with the support for non-latin characters. 


I have a same problem
I have a csv file like
```
实体,属性,值
胶饴,描述,别名: 饴糖、畅糖、畅、软糖。
词条,描述,词条（拼音：cí tiáo）也叫词目，是辞书学用语，指收列的词语及其释文。
词条,标签,文化
红色食品,描述,红色食品是指食品为红色、橙红色或棕红色的食品。
红色食品,中文名,红色食品
红色食品,是否含防腐剂,否
红色食品,主要食用功效,预防感冒，缓解疲劳
红色食品,适宜人群,全部人群
红色食品,用途,增强表皮细胞再生和防止皮肤衰老
```

when I loading data using vaex,the error throw
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-f05a9cfc1ef2> in <module>
----> 1 df = vaex.from_csv("/home/kevin/Downloads/34134795_princelai/ownthink_v2.csv",chunk_size=2000000,convert="bigdata.hdf5")

/opt/venv/py38/lib/python3.8/site-packages/vaex/__init__.py in from_csv(filename_or_buffer, copy_index, chunk_size, convert, **kwargs)
    498             # make it memory efficient by default
    499             chunk_size = 5_000_000
--> 500         return _from_csv_convert_and_read(filename_or_buffer=filename_or_buffer, copy_index=copy_index,
    501                                           maybe_convert_path=convert, chunk_size=chunk_size, **kwargs)
    502 

/opt/venv/py38/lib/python3.8/site-packages/vaex/__init__.py in _from_csv_convert_and_read(filename_or_buffer, copy_index, maybe_convert_path, chunk_size, **kwargs)
    538         df = from_pandas(df_pandas, copy_index=copy_index)
    539         filename_hdf5 = _convert_name(csv_path, suffix='_chunk%d' % i)
--> 540         df.export_hdf5(filename_hdf5, shuffle=False)
    541         converted_paths.append(filename_hdf5)
    542         logger.info('saved chunk #%d to %s' % (i, filename_hdf5))

/opt/venv/py38/lib/python3.8/site-packages/vaex/dataframe.py in export_hdf5(self, path, column_names, byteorder, shuffle, selection, progress, virtual, sort, ascending)
   5802         """
   5803         import vaex.export
-> 5804         vaex.export.export_hdf5(self, path, column_names, byteorder, shuffle, selection, progress=progress, virtual=virtual, sort=sort, ascending=ascending)
   5805 
   5806     def export_fits(self, path, column_names=None, shuffle=False, selection=False, progress=None, virtual=True, sort=None, ascending=True):

/opt/venv/py38/lib/python3.8/site-packages/vaex/export.py in export_hdf5(dataset, path, column_names, byteorder, shuffle, selection, progress, virtual, sort, ascending)
    310     kwargs = locals()
    311     import vaex.hdf5.export
--> 312     vaex.hdf5.export.export_hdf5(**kwargs)
    313 
    314 

/opt/venv/py38/lib/python3.8/site-packages/vaex/hdf5/export.py in export_hdf5(dataset, path, column_names, byteorder, shuffle, selection, progress, virtual, sort, ascending)
    162                 # else:
    163 
--> 164                 byte_length = dataset[column_name].str.byte_length().sum(selection=selection)
    165                 if byte_length > max_int32:
    166                     dtype_indices = 'i8'

/opt/venv/py38/lib/python3.8/site-packages/vaex/expression.py in sum(self, binby, limits, shape, selection, delay, progress)
    516         del kwargs['self']
    517         kwargs['expression'] = self.expression
--> 518         return self.ds.sum(**kwargs)
    519 
    520     def mean(self, binby=[], limits=None, shape=default_shape, selection=False, delay=False, progress=None):

/opt/venv/py38/lib/python3.8/site-packages/vaex/dataframe.py in sum(self, expression, binby, limits, shape, selection, delay, progress, edges, array_type)
    872         :return: {return_stat_scalar}
    873         """
--> 874         return self._compute_agg('sum', expression, binby, limits, shape, selection, delay, edges, progress, array_type=array_type)
    875         @delayed
    876         def finish(*sums):

/opt/venv/py38/lib/python3.8/site-packages/vaex/dataframe.py in _compute_agg(self, name, expression, binby, limits, shape, selection, delay, edges, progress, extra_expressions, array_type)
    623         for expression in expressions:
    624             if expression and expression != "*":
--> 625                 self.validate_expression(expression)
    626         if not hasattr(self.local, '_aggregator_nest_count'):
    627             self.local._aggregator_nest_count = 0

/opt/venv/py38/lib/python3.8/site-packages/vaex/dataframe.py in validate_expression(self, expression)
   2899         funcs = set(expression_namespace.keys())  | set(self.functions.keys())
   2900         try:
-> 2901             return vaex.expresso.validate_expression(expression, vars, funcs)
   2902         except NameError as e:
   2903             raise NameError(str(e)) from None

/opt/venv/py38/lib/python3.8/site-packages/vaex/expresso.py in validate_expression(expr, variable_set, function_set, names)
     76             raise ValueError("expected an expression got a %r" %
     77                              type(node.body))
---> 78         validate_expression(first_expr.value, variable_set,
     79                             function_set, names)
     80     elif isinstance(expr, _ast.BinOp):

/opt/venv/py38/lib/python3.8/site-packages/vaex/expresso.py in validate_expression(expr, variable_set, function_set, names)
    108         last_func = expr
    109         for arg in expr.args:
--> 110             validate_expression(arg, variable_set, function_set, names)
    111         for arg in expr.keywords:
    112             validate_expression(arg, variable_set, function_set, names)

/opt/venv/py38/lib/python3.8/site-packages/vaex/expresso.py in validate_expression(expr, variable_set, function_set, names)
     91             raise ValueError("Unary operator not allowed: %r" % expr.op)
     92     elif isinstance(expr, _ast.Name):
---> 93         validate_id(expr.id)
     94         if expr.id not in variable_set:
     95             matches = difflib.get_close_matches(expr.id, list(variable_set))

/opt/venv/py38/lib/python3.8/site-packages/vaex/expresso.py in validate_id(id)
    570     for char in id:
    571         if char not in valid_id_characters:
--> 572             raise ValueError("invalid character %r in id %r" % (char, id))

ValueError: invalid character '实' in id '实体'

```
Early feedback: this seems to be a problem still in master, and occurs when exporting the data (at least the latest issue).


Resolving patch:
diff --git a/packages/vaex-core/vaex/expresso.py b/packages/vaex-core/vaex/expresso.py
--- a/packages/vaex-core/vaex/expresso.py
+++ b/packages/vaex-core/vaex/expresso.py
@@ -97,7 +97,6 @@ def validate_expression(expr, variable_set, function_set=[], names=None):
         else:
             raise ValueError("Unary operator not allowed: %r" % expr.op)
     elif isinstance(expr, _ast.Name):
-        validate_id(expr.id)
         if expr.id not in variable_set:
             matches = difflib.get_close_matches(expr.id, list(variable_set))
             msg = "Column or variable %r does not exist." % expr.id
@@ -612,9 +611,3 @@ def node_to_string(node, pretty=False):
 def validate_func(name, function_set):
     if name.id not in function_set:
         raise NameError("function %r is not defined" % name.id)
-
-
-def validate_id(id):
-    for char in id:
-        if char not in valid_id_characters:
-            raise ValueError("invalid character %r in id %r" % (char, id))


Test patch:
diff --git a/tests/expresso_test.py b/tests/expresso_test.py
--- a/tests/expresso_test.py
+++ b/tests/expresso_test.py
@@ -65,3 +65,6 @@ def test_lists():
 def test_validate():
     validate_expression('x + 1', {'x'})
     validate_expression('x == "1"', {'x'})
+
+def test_unicode():
+    validate_expression('实体 + 1', {'实体'})