Repository: getredash/redash

Issue:
Error when running CSV download with InfluxDB as data source
Below is the error log for the Redash application.

Downloading CSV with InfluxDB as the data source causes an error.
The error log shows the following error
Since the test connection succeeds, we believe it is an application anomaly in Redash.

```
server_1            | KeyError: 'type'
server_1            | [2023-07-07 05:01:02,196][PID:9087][ERROR][redash.app] Exception on /api/queries/71/results/2093.csv [GET]
server_1            | Traceback (most recent call last):
server_1            |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
server_1            |     rv = self.dispatch_request()
server_1            |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
server_1            |     return self.view_functions[rule.endpoint](**req.view_args)
server_1            |   File "/usr/local/lib/python3.7/site-packages/flask_restful/__init__.py", line 458, in wrapper
server_1            |     resp = resource(*args, **kwargs)
server_1            |   File "/usr/local/lib/python3.7/site-packages/flask_login/utils.py", line 261, in decorated_view
server_1            |     return func(*args, **kwargs)
server_1            |   File "/usr/local/lib/python3.7/site-packages/flask/views.py", line 89, in view
server_1            |     return self.dispatch_request(*args, **kwargs)
server_1            |   File "/app/redash/handlers/base.py", line 33, in dispatch_request
server_1            |     return super(BaseResource, self).dispatch_request(*args, **kwargs)
server_1            |   File "/usr/local/lib/python3.7/site-packages/flask_restful/__init__.py", line 573, in dispatch_request
server_1            |     resp = meth(*args, **kwargs)
server_1            |   File "/app/redash/permissions.py", line 71, in decorated
server_1            |     return fn(*args, **kwargs)
server_1            |   File "/app/redash/handlers/query_results.py", line 409, in get
server_1            |     response = response_builders[filetype](query_result)
server_1            |   File "/app/redash/handlers/query_results.py", line 439, in make_csv_response
server_1            |     serialize_query_result_to_dsv(query_result, ","), 200, headers
server_1            |   File "/app/redash/serializers/query_result.py", line 87, in serialize_query_result_to_dsv
server_1            |     fieldnames, special_columns = _get_column_lists(query_data["columns"] or [])
server_1            |   File "/app/redash/serializers/query_result.py", line 68, in _get_column_lists
server_1            |     if col["type"] == col_type:
server_1            | KeyError: 'type'
```
Below is the JSON data when CSV download fails.
(Some parts are excerpted due to the length of the file.)

```
{
  "query_result": {
    "id": 2093,
    "query": "SELECT\n    SUM(d3) / 60\nFROM\n    charger \nWHERE\n    chg_id = '1'\n    AND time >= '2023-04-30T15:00:00Z'\n    AND time < '2023-05-03T15:00:00Z'\nGROUP BY chg_id, time(15m) fill(linear)\nlimit 10;",
    "data": {
      "columns": [{ "name": "time" }, { "name": "sum" }, { "name": "chg_id" }],
      "rows": [
        { "time": "2023-04-30T15:00:00Z", "sum": 0, "chg_id": "1" },
      ]
    }
  }
}
```

Below is the JSON data for a successful CSV download.

```
      "columns": [
        { "name": "id", "friendly_name": "id", "type": "integer" },
        {
          "name": "project_id",
          "friendly_name": "project_id",
          "type": "integer"
        },
        { "name": "name", "friendly_name": "name", "type": "string" },
        { "name": "filters", "friendly_name": "filters", "type": null },
        { "name": "user_id", "friendly_name": "user_id", "type": "integer" },
        {
          "name": "column_names",
          "friendly_name": "column_names",
          "type": null
        },
        {
```
In the normal case, an element is generated for each column,
In the case of failed data, three columns made up one element.
Therefore, the required type element was missing, resulting in a program error.
The following is a part of the program that caused the error.
```
    for col in columns:
        fieldnames.append(col["name"])

        for col_type in special_types.keys():
            if col["type"] == col_type:
                special_columns[col["name"]] = special_types[col_type]
```
This is just a guess, I am guessing, but it seems to me that the SQL results being executed against InfluxDB are special and are not being converted properly in Redash, which is what is causing the problem.

All text translated by Google Translate.


Discussion:
That's probably a useful bug report, but it's missing what the version of Redash this is happening with is. (?)
oh sorry.

The Version is Version: 10.0.0 (9c928bd1)

Thank you 
Thanks, that does sound like a bug we should look at. :smile:
@justinclift It looks like that InfluxDB v1 REST API and InfluxDBClient don't return a column type. So, the Query Runner for InfluxDB v1 doesn't return a column type.

https://github.com/getredash/redash/blob/3d32c55531d0b2148a51b2ac4201feebb495d818/redash/query_runner/influx_db.py#L45

My option is to take the value from the first row of the query results returned by InfluxDBClient and derive the column type from the Python type of that value. Can I create PR for this ?
@masayuki038 That sounds like a decent idea, and a reasonable foundation to possibly extend later on, if we need to adjust/improve the type detection. :smile:

Resolving patch:
diff --git a/redash/query_runner/influx_db.py b/redash/query_runner/influx_db.py
--- a/redash/query_runner/influx_db.py
+++ b/redash/query_runner/influx_db.py
@@ -1,6 +1,12 @@
 import logging
 
-from redash.query_runner import BaseQueryRunner, register
+from redash.query_runner import (
+    TYPE_FLOAT,
+    TYPE_INTEGER,
+    TYPE_STRING,
+    BaseQueryRunner,
+    register,
+)
 from redash.utils import json_dumps
 
 logger = logging.getLogger(__name__)
@@ -14,25 +20,36 @@
     enabled = False
 
 
+TYPES_MAP = {
+    str: TYPE_STRING,
+    int: TYPE_INTEGER,
+    float: TYPE_FLOAT,
+}
+
+
+def _get_type(value):
+    return TYPES_MAP.get(type(value), TYPE_STRING)
+
+
 def _transform_result(results):
-    result_columns = []
+    column_names = []
     result_rows = []
 
     for result in results:
         for series in result.raw.get("series", []):
             for column in series["columns"]:
-                if column not in result_columns:
-                    result_columns.append(column)
+                if column not in column_names:
+                    column_names.append(column)
             tags = series.get("tags", {})
             for key in tags.keys():
-                if key not in result_columns:
-                    result_columns.append(key)
+                if key not in column_names:
+                    column_names.append(key)
 
     for result in results:
         for series in result.raw.get("series", []):
             for point in series["values"]:
                 result_row = {}
-                for column in result_columns:
+                for column in column_names:
                     tags = series.get("tags", {})
                     if column in tags:
                         result_row[column] = tags[column]
@@ -42,7 +59,12 @@ def _transform_result(results):
                         result_row[column] = value
                 result_rows.append(result_row)
 
-    return json_dumps({"columns": [{"name": c} for c in result_columns], "rows": result_rows})
+    if len(result_rows) > 0:
+        result_columns = [{"name": c, "type": _get_type(result_rows[0][c])} for c in result_rows[0].keys()]
+    else:
+        result_columns = [{"name": c, "type": TYPE_STRING} for c in column_names]
+
+    return json_dumps({"columns": result_columns, "rows": result_rows})
 
 
 class InfluxDB(BaseQueryRunner):


Test patch:
diff --git a/tests/query_runner/test_influx_db.py b/tests/query_runner/test_influx_db.py
new file mode 100644
--- /dev/null
+++ b/tests/query_runner/test_influx_db.py
@@ -0,0 +1,58 @@
+import json
+
+from influxdb.resultset import ResultSet
+
+from redash.query_runner import (
+    TYPE_FLOAT,
+    TYPE_INTEGER,
+    TYPE_STRING,
+)
+from redash.query_runner.influx_db import _transform_result
+
+raw = {
+    "series": [
+        {
+            "name": "typetest",
+            "columns": ["time", "k1", "v1", "v2"],
+            "values": [
+                ["2023-10-06T13:30:51.323358136Z", "foo", 0.5, 2],
+                ["2023-10-06T13:31:08.882953339Z", "bar", 0.6, 4],
+            ],
+        }
+    ]
+}
+
+raw_no_rows = {"series": [{"name": "typetest", "columns": ["time", "k1", "v1", "v2"], "values": []}]}
+
+
+def test_influxdb_result_types_with_rows():
+    result = ResultSet(raw)
+    transformed = _transform_result([result])
+    expected = {
+        "columns": [
+            {"name": "time", "type": TYPE_STRING},
+            {"name": "k1", "type": TYPE_STRING},
+            {"name": "v1", "type": TYPE_FLOAT},
+            {"name": "v2", "type": TYPE_INTEGER},
+        ],
+        "rows": [
+            {"k1": "foo", "time": "2023-10-06T13:30:51.323358136Z", "v1": 0.5, "v2": 2},
+            {"k1": "bar", "time": "2023-10-06T13:31:08.882953339Z", "v1": 0.6, "v2": 4},
+        ],
+    }
+    assert json.loads(transformed) == expected
+
+
+def test_influxdb_result_types_with_no_rows_are_string():
+    result = ResultSet(raw_no_rows)
+    transformed = _transform_result([result])
+    expected = {
+        "columns": [
+            {"name": "time", "type": TYPE_STRING},
+            {"name": "k1", "type": TYPE_STRING},
+            {"name": "v1", "type": TYPE_STRING},
+            {"name": "v2", "type": TYPE_STRING},
+        ],
+        "rows": [],
+    }
+    assert json.loads(transformed) == expected