Repository: ibis-project/ibis

Issue:
bug (Impala): "date" type not recognized when trying to read Hive table
A Hive table I'm trying to read has columns with the "DATE" type (valid type as per [the Impala docs](https://impala.apache.org/docs/build/html/topics/impala_date.html)).

When trying to run a simple expression (`table = connection.table('my_db.my_table'`), I get the following traceback:

```Exception                                 Traceback (most recent call last)
Input In [9], in <cell line: 1>()
----> 1 hive_conn.table('tableau_di_tables.ds_consumer_travel_orders')

File ~/miniconda3/envs/ds-hotel-ranking/lib/python3.10/site-packages/ibis/backends/base/sql/__init__.py:44, in BaseSQLBackend.table(self, name, database)
     29 """Construct a table expression.
     30 
     31 Parameters
   (...)
     41     Table expression
     42 """
     43 qualified_name = self._fully_qualified_name(name, database)
---> 44 schema = self.get_schema(qualified_name)
     45 node = self.table_class(qualified_name, schema, self)
     46 return self.table_expr_class(node)

File ~/miniconda3/envs/ds-hotel-ranking/lib/python3.10/site-packages/ibis/backends/impala/__init__.py:482, in Backend.get_schema(self, table_name, database)
    479 pairs = [row[:2] for row in self.con.fetchall(query)]
    481 names, types = zip(*pairs)
--> 482 ibis_types = [udf.parse_type(type.lower()) for type in types]
    483 return sch.Schema(names, ibis_types)

File ~/miniconda3/envs/ds-hotel-ranking/lib/python3.10/site-packages/ibis/backends/impala/__init__.py:482, in <listcomp>(.0)
    479 pairs = [row[:2] for row in self.con.fetchall(query)]
    481 names, types = zip(*pairs)
--> 482 ibis_types = [udf.parse_type(type.lower()) for type in types]
    483 return sch.Schema(names, ibis_types)

File ~/miniconda3/envs/ds-hotel-ranking/lib/python3.10/site-packages/ibis/backends/impala/udf.py:310, in parse_type(t)
    308         return ValueError(t)
    309 else:
--> 310     raise Exception(t)

Exception: date
```


Looking at the source code it seems that "date" type is not in the `_impala_to_ibis_type` dict in `udf.py`.

Is it possible to include the date type here? It seems like a one-to-one equivalency with the ibis type.

 I can try and submit a pull request for this although I don't have much experience with contributing!


Discussion:
Hey @indylec!  Thanks for the report!  

I think you've diagnosed the issue correctly (although I haven't used the `impala` backend much at all).  Try out your fix locally and if that resolves the issue then we'd certainly welcome a pull request with the fix (and we're happy to help walk you through any rough patches in the contribution process).
Hi @gforsyth, 

Ok, I'll try it out locally and attempt a pull request - stay tuned.

/take
Hi @gforsyth, 

So the fix works locally, I also had to add a key-value pair to the `_HS2_TTypeId_to_dtype` dict in `impala/__init__.py` (`'DATE':'datetime64[ns]'`).
The table I need to read also has a couple of undefined columns (show up as "VOID" type in dBeaver) - I have also added the appropriate key-value pairs for this (`'void':'null'` in `udf.py` and `'void':None` in `__init__.py`)

Now to the (first) rough patch:

The core test suite passes ( `3116 passed, 6016 deselected, 4 xfailed in 18.77s`), however the Impala subset fails because the tests are not able to connect to the expected test databases (`1 failed, 280 passed, 8559 deselected, 26 xfailed, 805 errors in 122.58s`).

Any guidance on how to set this up would be welcome, alternatively I can make the commit to my fork and someone else can run the Impala tests?

Thanks!



@indylec If you want to push up the PR, I can help you get through the test suite.

Resolving patch:
diff --git a/ibis/backends/impala/__init__.py b/ibis/backends/impala/__init__.py
--- a/ibis/backends/impala/__init__.py
+++ b/ibis/backends/impala/__init__.py
@@ -67,6 +67,8 @@
     'BINARY': 'object',
     'VARCHAR': 'object',
     'CHAR': 'object',
+    'DATE': 'datetime64[ns]',
+    'VOID': None,
 }
 
 
diff --git a/ibis/backends/impala/udf.py b/ibis/backends/impala/udf.py
--- a/ibis/backends/impala/udf.py
+++ b/ibis/backends/impala/udf.py
@@ -346,4 +346,6 @@ def _ibis_string_to_impala(tval):
     'char': 'string',
     'timestamp': 'timestamp',
     'decimal': 'decimal',
+    'date': 'date',
+    'void': 'null',
 }


Test patch: