Repository: ibis-project/ibis (Python)

Issue:
bug(clickhouse):  SQL is generated with `ARRAYJOIN` function instead of `arrayJoin`
### What happened?

Hi, thanks for this great project!

When using a subquery to filter a column the clickhouse backend generated the following SQL code:

```sql
SELECT
  t0.id,
  t0.longitude,
  t0.latitude
FROM osm.node_view AS t0
WHERE
  t0.id IN (
    SELECT
      t5.node_id
    FROM (
      SELECT
        t4.id AS way_id,
        t4.tag_key,
        t4.tag_value,
        CAST(t4.nodes.2 AS Nullable(Int64)) AS node_id
      FROM (
        SELECT
          t3.id,
          t3.tag_key,
          t3.tag_value,
          ARRAYJOIN(t3.nodes) AS nodes --HERE
        FROM (
          SELECT
            t2.id,
            t2.tag_key,
            t2.tag_value,
            t2.nodes
          FROM (
            SELECT
              t1.id,
              CAST(t1.tags.1 AS Nullable(TEXT)) AS tag_key,
              CAST(t1.tags.2 AS Nullable(TEXT)) AS tag_value,
              t1.nodes
            FROM (
              SELECT
                t0.id,
                ARRAYJOIN(t0.tags) AS tags, --HERE
                t0.nodes
              FROM osm.way_view AS t0
            ) AS t1
          ) AS t2
          WHERE
            t2.tag_key = 'amenity'
            AND t2.tag_value IN ('bar', 'biergarten', 'cafe', 'fast_food', 'food_court', 'ice_cream', 'pub', 'restaurant')
        ) AS t3
      ) AS t4
    ) AS t5
  )
```

The function `ARRAYJOIN` is not available in Clickhouse, that expects `arrayJoin`.  Replacing `ARRAYJOIN` for `arrayJoin` successfully executes the query and provides the correct result.

Looking at the Clickhouse backend code, it seems to be using the correct function: https://github.com/ibis-project/ibis/blob/eb4602fada06b5f2b64d496c42f769598ef5d99e/ibis/backends/clickhouse/compiler/values.py#L1058

### What version of ibis are you using?

5.1.0

### What backend(s) are you using, if any?

Clickhouse

### Relevant log output

```sh
DB::Exception: Unknown function ARRAYJOIN. Maybe you meant: ['arrayJoin','arrayMin']: While processing id, ARRAYJOIN(tags) AS tags, nodes: While processing id IN ((SELECT t5.node_id FROM (SELECT t4.id AS way_id, t4.tag_key, t4.tag_value, CAST(t4.nodes.2, 'Nullable(Int64)') AS node_id FROM (SELECT t3.id, t3.tag_key, t3.tag_value, arrayJoin(t3.nodes) AS nodes FROM (SELECT t2.id, t2.tag_key, t2.tag_value, t2.nodes FROM (SELECT t1.id, CAST(t1.tags.1, 'Nullable(TEXT)') AS tag_key, CAST(t1.tags.2, 'Nullable(TEXT)') AS tag_value, t1.nodes FROM (SELECT t0.id, ARRAYJOIN(t0.tags) AS tags, t0.nodes FROM osm.way_view AS t0) AS t1) AS t2 WHERE (t2.tag_key = 'amenity') AND (t2.tag_value IN ('bar', 'biergarten', 'cafe', 'fast_food', 'food_court', 'ice_cream', 'pub', 'restaurant'))) AS t3) AS t4) AS t5) AS _subquery43)
```


### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion: @guillemborrell Welcome 👋🏻

Thanks for opening up an issue.

It looks like you may have a slightly older version of sqlglot that had this bug.

Can you show the version of `sqlglot` that you have installed?

At the time 5.1.0 was published the `poetry.lock` file had version 11.5.2 (you may have a different version because we avoid pinning our dependencies).

Testing with sqlglot 11.5.2 at the 5.1.0 commit (374490ed643f750b2a0827c2951b45a7160fc0d1) shows that `arrayJoin` is correctly preserved:

```
In [11]: import ibis

In [12]: from ibis import _

In [13]: t = ibis.table(dict(id="int64", tags="array<struct<key: string, value: string>>", nodes="array<string>"), name
    ...: ="way_view")

In [14]: expr = t.mutate(tags=_.tags.unnest()).select("id", key=_.tags["key"], value=_.tags["value"], nodes=_.nodes)

In [15]: print(ibis.clickhouse.compile(expr))
SELECT
  t1.id,
  CAST(t1.tags.1 AS Nullable(TEXT)),
  CAST(t1.tags.2 AS Nullable(TEXT)),
  t1.nodes
FROM (
  SELECT
    t0.id,
    arrayJoin(t0.tags) AS tags,
    t0.nodes
  FROM way_view AS t0
) AS t1

In [16]: import sqlglot as sg

In [17]: sg.__version__
Out[17]: '11.5.2'
```

Can you also show the ibis code that's generating this SQL, or a smaller expression that's reproducing the problem?

Thank you!
Thanks @cpcloud 

I was able to reproduce the bug with tables as you suggested:

```python
from ibis import clickhouse, _
import ibis

node_view = ibis.table(
    dict(
        id="int64",
        version="int32",
        timestamp="int64",
        changset="int64",
        uid="int32",
        user_sid="string",
        tags="!array<!struct<key: string, value: string>>", 
        longitude="float64",
        latitude="float64"
    ), 
    name="osm.node_view")

way_view = ibis.table(
    dict(
        id="int64",
        version="int32",
        timestamp="int64",
        changset="int64",
        uid="int32",
        user_sid="string",
        nodes="!array<!struct<index: int32, nodeId: int64>>",
        tags="!array<!struct<key: string, value: string>>"
    ),
    name="osm.way_view"
)

node_tags = (node_view
 .select(id = node_view.id,
         tags = node_view.tags.unnest())
 .select(id = _.id,
         tag_key = _[1]['key'],
         tag_value = _[1]['value']
         )
 .select([_.id, _.tag_key, _.tag_value])
 .filter([_.tag_key == 'amenity',
          _.tag_value.isin(('bar', 'biergarten', 'cafe', 'fast_food', 'food_court', 'ice_cream', 'pub', 'restaurant'))])
)

way_node_tags = (
    way_view
    .select(id = way_view.id,
            tags = way_view.tags.unnest(),
            nodes = way_view.nodes)
    .select(id = _.id,
            tag_key = _[1]['key'],
            tag_value = _[1]['value'],
            nodes = _.nodes
            )
    .select([_.id, _.tag_key, _.tag_value, _.nodes])
    .filter([_.tag_key == 'amenity',
             _.tag_value.isin(('bar', 'biergarten', 'cafe', 'fast_food', 'food_court', 'ice_cream', 'pub', 'restaurant'))])
    .select([_.id, _.tag_key, _.tag_value, _.nodes.unnest()])
    .select(way_id = _.id,
            tag_key = _.tag_key,
            tag_value = _.tag_value,
            node_id = _.nodes['nodeId'])
)

way_node_tags_lonlat = (
    node_view
    .select([node_view.id, node_view.longitude, node_view.latitude])
    .filter(node_view.id.isin(way_node_tags.node_id))
)

clickhouse.compile(way_node_tags_lonlat)
```

I aligned the versions of ibis and sqlglot to 5.1.0 and 11.5.2 respectively.
Excellent, I am able to reproduce. Slated for 6.0!



Fix (patch):
diff --git a/ibis/backends/clickhouse/compiler/values.py b/ibis/backends/clickhouse/compiler/values.py
index 52862e8ae479..46b14c3581ff 100644
--- a/ibis/backends/clickhouse/compiler/values.py
+++ b/ibis/backends/clickhouse/compiler/values.py
@@ -850,9 +850,9 @@ def tr(op, *, cache, **kw):
             # annoying to detect so we let it through to enable the
             # uncorrelated use case (pandas-style `.isin`)
             subquery = translate(options.to_expr().as_table().op(), {})
-            right_arg = f"({subquery})"
+            right_arg = f"({_sql(subquery)})"
         else:
-            right_arg = translate_val(options, cache=cache, **kw)
+            right_arg = _sql(translate_val(options, cache=cache, **kw))
 
         # we explicitly do NOT parenthesize the right side because it doesn't
         # make sense to do so for Sequence operations