internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ibis-project/ibis

Issue:
bug: non-string Categorical from pandas crash memtable()
### What happened?

```python
import ibis
import pandas as pd

df = pd.DataFrame(
    {
        "cat_string": pd.Series(["a", "b", "c"], dtype="category"),
        "cat_int": pd.Series([1, 2, 3], dtype="category"),
    }
)
ibis.memtable(df).execute()
# ArrowTypeError: ('Expected a string or bytes dtype, got int64', 'Conversion failed for column cat_int with type category')
```

### What version of ibis are you using?

main (5998143f09add788bb569b9e3ea091240ba00a26, Jan 22, 2024)

### What backend(s) are you using, if any?

duckdb

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:

Resolving patch:
diff --git a/ibis/formats/pandas.py b/ibis/formats/pandas.py
--- a/ibis/formats/pandas.py
+++ b/ibis/formats/pandas.py
@@ -33,7 +33,9 @@ def to_ibis(cls, typ, nullable=True):
         elif pdt.is_datetime64_dtype(typ):
             return dt.Timestamp(nullable=nullable)
         elif isinstance(typ, pdt.CategoricalDtype):
-            return dt.String(nullable=nullable)
+            if typ.categories is None or pdt.is_string_dtype(typ.categories):
+                return dt.String(nullable=nullable)
+            return cls.to_ibis(typ.categories.dtype, nullable=nullable)
         elif pdt.is_extension_array_dtype(typ):
             if _has_arrow_dtype and isinstance(typ, pd.ArrowDtype):
                 return PyArrowType.to_ibis(typ.pyarrow_dtype, nullable=nullable)


Test patch:
diff --git a/ibis/formats/tests/test_pandas.py b/ibis/formats/tests/test_pandas.py
--- a/ibis/formats/tests/test_pandas.py
+++ b/ibis/formats/tests/test_pandas.py
@@ -103,6 +103,8 @@ def test_dtype_from_pandas_arrow_list_dtype():
             dt.Timestamp("US/Eastern"),
         ),
         (pd.CategoricalDtype(), dt.String()),
+        (pd.CategoricalDtype(["a", "b", "c"]), dt.String()),
+        (pd.CategoricalDtype(np.array([1, 2, 3], dtype="int64")), dt.int64),
         (pd.Series([], dtype="string").dtype, dt.String()),
     ],
     ids=str,
Repository: ibis-project/ibis

Issue:
bug: `ibis.examples.___.fetch()` can fail if Google Cloud SDK is installed
### What happened?

When I have the [Google Cloud SDK](https://cloud.google.com/sdk/) installed and configured, fetching example data from GCS fails with
```
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})
```
This error goes away if I [uninstall the Google Cloud SDK](https://cloud.google.com/sdk/docs/uninstall-cloud-sdk).

Can we configure the example data fetch function to always send the GCS request anonymously, ignoring any local Google Cloud SDK configuration?

### What version of ibis are you using?

8.0.0.dev

### What backend(s) are you using, if any?

_No response_

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
Sounds like a misconfigured client. We use gcs and other gcp services without having any issues like this. I'm not particularly inclined to change it without more information about why it's failing for you.
Fair enough, but would there be any reason _not_ to set `token='anon'` when connecting to GCS to fetch example data? https://gcsfs.readthedocs.io/en/latest/api.html#gcsfs.core.GCSFileSystem

Resolving patch:
diff --git a/ibis/examples/__init__.py b/ibis/examples/__init__.py
--- a/ibis/examples/__init__.py
+++ b/ibis/examples/__init__.py
@@ -118,7 +118,9 @@ def _get_metadata():
 def _get_board():
     import pins
 
-    return pins.board_gcs(_BUCKET)
+    return pins.board(
+        "gcs", _BUCKET, storage_options={"cache_timeout": 0, "token": "anon"}
+    )
 
 
 @functools.cache


Test patch:
Repository: ibis-project/ibis

Issue:
docs: `row_number` always starts at 0
### Please describe the issue

its unclear from the documentation that `row_number` will always start at 0 for all backends

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:

Resolving patch:
diff --git a/ibis/expr/api.py b/ibis/expr/api.py
--- a/ibis/expr/api.py
+++ b/ibis/expr/api.py
@@ -1240,6 +1240,10 @@ def ntile(buckets: int | ir.IntegerValue) -> ir.IntegerColumn:
 def row_number() -> ir.IntegerColumn:
     """Return an analytic function expression for the current row number.
 
+    ::: {.callout-note}
+    `row_number` is normalized across backends to start at 0
+    :::
+
     Returns
     -------
     IntegerColumn


Test patch:
Repository: ibis-project/ibis

Issue:
bug: can't preview table containing UUIDs
### What happened?

```python
import ibis
ibis.options.interactive = True

t = ibis.memtable(
    {
        "x": [1, 2, 3],
    }
)
con = ibis.duckdb.connect()
t = con.create_table("t", t, overwrite=True)
t = t.sql("SELECT *, uuid() as id FROM t")
t
```
After the .sql() call, there is to interactive preview showing up. If I remove the .sql() line, it does show up

<img width="421" alt="image" src="https://github.com/ibis-project/ibis/assets/10820686/26d952b9-97ef-4ae2-a4f9-f3019f7b4e85">

<img width="399" alt="image" src="https://github.com/ibis-project/ibis/assets/10820686/d41701b1-6bff-4a20-ae9b-57f54ff3328a">



### What version of ibis are you using?

8.0.0

### What backend(s) are you using, if any?

duckdb

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
Thanks for the issue. I suspect we're not handling the UUID type in formatting.
oh, that is totally all it it. changing the sql to "SELECT *, 5 as id FROM t" makes the preview show up
ah, it is because arrow doesn't support uuids. So in `to_rich_table()` when we execute the expression to_pyarrow, that fails.
Right, I think a `format_values` overload for the UUID dtype might do the trick.
Naw, there already is an overload for UUID, it never even gets to that function.

changing [this line](https://github.com/ibis-project/ibis/blob/03acbd10ab55e6cdc9759682eafb79f13c4b4950/ibis/expr/types/pretty.py#L265)  to

```python
    head = table.limit(nrows + 1)
    # pyarrow can't handle UUID types, so we need to convert them to strings for now
    head = head.mutate(s.across(s.of_type("uuid"), lambda x: x.cast("string")))
    result = head.to_pyarrow()
```

makes it work.

What do you think, is that actually a god resolution? Don't really want to go into implementing uuids for pyarrow. Could make UUIDs always get cast to strings during the .to_pyarrow() call? But that is hiding something, users might want to have to do that cast explicitly like I am here.
Hm, actually we _do_ have a UUID overload for `format_values`. This is not an issue in PyArrow, it's an ibis issue.
@NickCrews Neither your solution nor mucking around in pyarrow is the right solution here 😅 
haha love it when those are the options :)

Resolving patch:
diff --git a/ibis/formats/pyarrow.py b/ibis/formats/pyarrow.py
--- a/ibis/formats/pyarrow.py
+++ b/ibis/formats/pyarrow.py
@@ -90,6 +90,7 @@ def __arrow_ext_scalar_class__(self):
     dt.Unknown: pa.string(),
     dt.MACADDR: pa.string(),
     dt.INET: pa.string(),
+    dt.UUID: pa.string(),
 }
 
 


Test patch:
diff --git a/ibis/formats/tests/test_pyarrow.py b/ibis/formats/tests/test_pyarrow.py
--- a/ibis/formats/tests/test_pyarrow.py
+++ b/ibis/formats/tests/test_pyarrow.py
@@ -130,6 +130,7 @@ def test_dtype_from_nullable_list_type(value_nullable, list_nullable):
             dt.Set(dt.String(nullable=False)),
             pa.list_(pa.field("item", pa.string(), nullable=False)),
         ),
+        (dt.uuid, pa.string()),
     ],
 )
 def test_ibis_exclusive_types(ibis_type, arrow_type):
Repository: ibis-project/ibis

Issue:
bug: `Scalar.isin(Column)` returns a Column, not a Scalar
### What happened?

```python
import ibis

needle = ibis.literal(2)
haystack = ibis.memtable({"x": [1, 2, 3]}).x
type(needle.isin(haystack))
# ibis.expr.types.logical.BooleanColumn
```

### What version of ibis are you using?

main

### What backend(s) are you using, if any?

_No response_

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:

Resolving patch:
diff --git a/ibis/expr/operations/subqueries.py b/ibis/expr/operations/subqueries.py
--- a/ibis/expr/operations/subqueries.py
+++ b/ibis/expr/operations/subqueries.py
@@ -53,7 +53,7 @@ class InSubquery(Subquery):
     needle: Value
 
     dtype = dt.boolean
-    shape = ds.columnar
+    shape = rlz.shape_like("needle")
 
     def __init__(self, rel, needle):
         if len(rel.schema) != 1:


Test patch:
diff --git a/ibis/tests/expr/test_value_exprs.py b/ibis/tests/expr/test_value_exprs.py
--- a/ibis/tests/expr/test_value_exprs.py
+++ b/ibis/tests/expr/test_value_exprs.py
@@ -1713,3 +1713,13 @@ def test_deferred_doesnt_convert_callables():
         b=t.b.split(",").filter(lambda pp: ~pp.isin(("word1", "word2")))
     )
     assert expr.equals(expected)
+
+
+def test_in_subquery_shape():
+    t = ibis.table([("a", "int64"), ("b", "string")])
+
+    expr = t.a.cast("string").isin(t.b)
+    assert expr.op().shape.is_columnar()
+
+    expr = ibis.literal(2).isin(t.a)
+    assert expr.op().shape.is_scalar()
Repository: ibis-project/ibis

Issue:
docs: explain you only need to close a conn.raw_sql() if you are running a query that returns results
### Please describe the issue

I was following the advise of https://ibis-project.org/how-to/extending/sql#backend.raw_sql, and closing the returned cursor.

But the actual SQL I was running was an `EXPLAIN SELECT ...` query. I was getting errors, eventually after a lot of headscratching I think I have found that I do NOT want to close the cursor after this query.

Possible actions:
1. be more precise with what sorts of SQL statements require the close.
2. make it so ALL queries require a close, so there is no distinction.
3. UX: make the returned object itself closable, so I don't have to do `from contextlib import closing` and I can just do `with conn.raw_sql(x) as cursor:`

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
@NickCrews Thanks for the issue!

I think option 1 is probably the best we can do here without a ton of additional effort.

> make it so ALL queries require a close, so there is no distinction.

This is out of the control of Ibis. There's no agreed-upon-by-all-backends criteria for whether a cursor needs to be closed. The documentation is best effort. I can certainly update it to reflect the variation.

> make the returned object itself closable

We're unlikely to do this because it would require wrapping the backend's cursor, and the point of `raw_sql` is to expose the backend's cursor directly, without interfering. The trade off is that users must now deal with any inconveniences brought on by the backend's driver.
To clarify, you should only need to close the cursor if you're doing a `SELECT` statement, or more generally a statement that can return values. DDL statements should **not** require closing the cursor.

Resolving patch:
diff --git a/ibis/config.py b/ibis/config.py
--- a/ibis/config.py
+++ b/ibis/config.py
@@ -147,7 +147,7 @@ class Options(Config):
         A callable to use when logging.
     graphviz_repr : bool
         Render expressions as GraphViz PNGs when running in a Jupyter notebook.
-    default_backend : Optional[ibis.backends.BaseBackend], default None
+    default_backend : Optional[ibis.backends.BaseBackend]
         The default backend to use for execution, defaults to DuckDB if not
         set.
     context_adjustment : ContextAdjustment


Test patch:
Repository: ibis-project/ibis

Issue:
bug: regression: slow to compute name of deeply nested expressions
### What happened?

```python
import ibis

x = ibis.literal(1)
for i in range(100):
    x = x + 1
    # if I hardcode the name, this is instantaneous
    # x = x.name(f"step_{i}")
x
```

If I ctrl-c this after a few seconds, I see see eg

```
File ~/code/ibis/ibis/expr/operations/core.py:89, in Value.name(self)
     86 @property
     87 def name(self) -> str:
     88     names = (arg.name for arg in self.__args__ if hasattr(arg, "name"))
---> 89     return f"{self.__class__.__name__}({', '.join(names)})"

File ~/code/ibis/ibis/expr/operations/core.py:88, in <genexpr>(.0)
     86 @property
     87 def name(self) -> str:
---> 88     names = (arg.name for arg in self.__args__ if hasattr(arg, "name"))
     89     return f"{self.__class__.__name__}({', '.join(names)})"

    [... skipping similar frames: <genexpr> at line 88 (97 times), Value.name at line 89 (97 times)]

File ~/code/ibis/ibis/expr/operations/core.py:89, in Value.name(self)
     86 @property
     87 def name(self) -> str:
     88     names = (arg.name for arg in self.__args__ if hasattr(arg, "name"))
---> 89     return f"{self.__class__.__name__}({', '.join(names)})"

File ~/code/ibis/ibis/expr/operations/core.py:88, in <genexpr>(.0)
     86 @property
     87 def name(self) -> str:
---> 88     names = (arg.name for arg in self.__args__ if hasattr(arg, "name"))
     89     return f"{self.__class__.__name__}({', '.join(names)})"
```

### What version of ibis are you using?

main 09b6adaeecf1b6388866856c795022cfca4b2679

### What backend(s) are you using, if any?

NA

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
Thanks!

If this is noticeably slow then it's probably recursing infinitely.
Ok, it's not infinite: reducing the number of iterations to 10 returns quickly.

Resolving patch:
diff --git a/ibis/expr/operations/core.py b/ibis/expr/operations/core.py
--- a/ibis/expr/operations/core.py
+++ b/ibis/expr/operations/core.py
@@ -85,7 +85,11 @@ def __coerce__(
     # TODO(kszucs): figure out how to represent not named arguments
     @property
     def name(self) -> str:
-        names = (arg.name for arg in self.__args__ if hasattr(arg, "name"))
+        names = (
+            name
+            for arg in self.__args__
+            if (name := getattr(arg, "name", None)) is not None
+        )
         return f"{self.__class__.__name__}({', '.join(names)})"
 
     @property


Test patch:
diff --git a/ibis/expr/tests/test_api.py b/ibis/expr/tests/test_api.py
--- a/ibis/expr/tests/test_api.py
+++ b/ibis/expr/tests/test_api.py
@@ -147,3 +147,12 @@ def test_implicit_coercion_of_null_literal(op):
 
     assert expr1.op() == expected
     assert expr2.op() == expected
+
+
+def test_nested_name_property():
+    x = ibis.literal(1)
+    n = 100
+    for _ in range(n):  # noqa: F402
+        x = x + 1
+
+    assert x.op().name.count("Add") == n
Repository: ibis-project/ibis

Issue:
bug: datafusion `to_pyarrow_batches` computes when source is a parquet file
### What happened?

When `to_pyarrow_batches` is invoked on a datafusion expr, whether or not it does compute before emitting a `RecordBatchReader` depends upon the source table type. I believe wrapping the `frame.collect` invocation inside of `to_pyarrow_batches` in a function that does `yield from` makes all paths the same.

When I run this snippet
```
path = "numeric.parquet"
table = ibis.datafusion.connect().register(path, "t").to_pyarrow()
print(table.shape)
sources = (table, path)
backends = ("duckdb", "datafusion")
for backend in backends:
    for source in sources:
        con = getattr(ibis, backend).connect()
        with Timer(f"type(source)={type(source)}, backend={backend}"):
            with Timer(f"register", level=1):
                t = con.register(source, "t")
            with Timer(f"to_pyarrow_batches", level=1):
                batches = t.to_pyarrow_batches()
```

I see the following output
```
(2260668, 109)
        register took:       40 ms
        to_pyarrow_batches took:             45 ms
type(source)=<class 'pyarrow.lib.Table'>, backend=duckdb took:       85 ms
        register took:       46 ms
        to_pyarrow_batches took:             34 ms
type(source)=<class 'str'>, backend=duckdb took:             80 ms
        register took:       70 ms
        to_pyarrow_batches took:             75 ms
type(source)=<class 'pyarrow.lib.Table'>, backend=datafusion took:          145 ms
        register took:        5 ms
        to_pyarrow_batches took:            940 ms
type(source)=<class 'str'>, backend=datafusion took:        946 ms
```

### What version of ibis are you using?

In [9]: ibis.__version__
Out[9]: '8.0.0'

### What backend(s) are you using, if any?

datafusion, duckdb

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:

Resolving patch:
diff --git a/ibis/backends/datafusion/__init__.py b/ibis/backends/datafusion/__init__.py
--- a/ibis/backends/datafusion/__init__.py
+++ b/ibis/backends/datafusion/__init__.py
@@ -509,9 +509,8 @@ def to_pyarrow_batches(
 
         struct_schema = schema.as_struct().to_pyarrow()
 
-        return pa.ipc.RecordBatchReader.from_batches(
-            schema.to_pyarrow(),
-            (
+        def make_gen():
+            yield from (
                 # convert the renamed + casted columns into a record batch
                 pa.RecordBatch.from_struct_array(
                     # rename columns to match schema because datafusion lowercases things
@@ -522,7 +521,11 @@ def to_pyarrow_batches(
                     .cast(struct_schema)
                 )
                 for batch in frame.collect()
-            ),
+            )
+
+        return pa.ipc.RecordBatchReader.from_batches(
+            schema.to_pyarrow(),
+            make_gen(),
         )
 
     def to_pyarrow(self, expr: ir.Expr, **kwargs: Any) -> pa.Table:


Test patch:
Repository: ibis-project/ibis

Issue:
docs: add RisingWave to the installs tab
### Please describe the issue

in `docs/_tabsets/instal.qmd`, add the RisingWave backend

I took a stab at this and could not get it to regenerate while rewriting another document, opening this for tracking (and if anyone has a tip -- simply adding a new line for it in the backends list of dictionaries didn't work w/ a `just docs-preview`)

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:

Resolving patch:
diff --git a/docs/backends_sankey.py b/docs/backends_sankey.py
new file mode 100644
--- /dev/null
+++ b/docs/backends_sankey.py
@@ -0,0 +1,85 @@
+from __future__ import annotations
+
+import plotly.graph_objects as go
+
+backend_categories = {
+    "SQL-generating": [
+        "BigQuery",
+        "ClickHouse",
+        "DataFusion",
+        "Druid",
+        "DuckDB",
+        "Exasol",
+        "Flink",
+        "Impala",
+        "MSSQL",
+        "MySQL",
+        "Oracle",
+        "PostgreSQL",
+        "PySpark",
+        "RisingWave",
+        "Snowflake",
+        "SQLite",
+        "Trino",
+    ],
+    "Expression-generating": ["Dask", "Polars"],
+    "Naïve execution": ["pandas"],
+}
+
+category_colors = {
+    "Ibis API": "#999999",
+    "Naïve execution": "#FF8C00",
+    "Expression-generating": "#6A5ACD",
+    "SQL-generating": "#3CB371",
+}
+
+nodes, links = [], []
+node_index = {}
+
+nodes.append({"label": "Ibis API", "color": category_colors["Ibis API"]})
+node_index["Ibis API"] = 0
+
+
+idx = 1
+for category, backends in backend_categories.items():
+    nodes.append({"label": category, "color": category_colors[category]})
+    node_index[category] = idx
+    links.append({"source": 0, "target": idx, "value": len(backends)})
+    idx += 1
+
+    for backend in backends:
+        if backend not in node_index:
+            nodes.append({"label": backend, "color": category_colors[category]})
+            node_index[backend] = idx
+            idx += 1
+        links.append(
+            {
+                "source": node_index[category],
+                "target": node_index[backend],
+                "value": 1,
+            }
+        )
+
+
+fig = go.Figure(
+    data=[
+        go.Sankey(
+            node=dict(
+                pad=20,
+                thickness=20,
+                line=dict(color="black", width=0.5),
+                label=[node["label"] for node in nodes],
+                color=[node["color"] for node in nodes],
+            ),
+            link=dict(
+                source=[link["source"] for link in links],
+                target=[link["target"] for link in links],
+                value=[link["value"] for link in links],
+            ),
+        )
+    ]
+)
+
+fig.update_layout(
+    title_text="Ibis backend types", font_size=14, margin=dict(l=30, r=30, t=80, b=30)
+)


Test patch:
Repository: ibis-project/ibis

Issue:
bug(bigquery): user-provided client's default job config is modified
### What happened?

Initialize a BigQuery (BQ) client:

```python
>>> from google.cloud import bigquery
>>> encryption_config = bigquery.EncryptionConfiguration(kms_key_name="projects/my-project/locations/global/myKeyRing/cryptoKeys/myKey")
>>> query_job_config = bigquery.QueryJobConfig(destination_encryption_configuration=encryption_config)
>>> bqclient = bigquery.Client(default_query_job_config=query_job_config)
>>> bqclient.default_query_job_config.destination_encryption_configuration
EncryptionConfiguration(projects/my-project/locations/global/myKeyRing/cryptoKeys/myKey)
>>>
```

Pass the BQ client to to ibis bigquery connection:

```python
>>> import ibis
>>> ibis.bigquery.connect(client=bqclient)
<ibis.backends.bigquery.Backend object at 0x7febe7bcfaf0>
```

Now the passed BQ client's default job config has been modified:

```python
>>> bqclient.default_query_job_config.destination_encryption_configuration
>>>
```

### What version of ibis are you using?

```python
>>> ibis.__version__
'7.1.0'
```

### What backend(s) are you using, if any?

BigQuery

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
I think this is where the bug is: https://github.com/ibis-project/ibis/blob/c2a9ba9a1dd9134d32ba696c92f2de50e99117f9/ibis/backends/bigquery/__init__.py#L436
Thanks for the report!

Let me see if I can remove this setting altogether without breaking anything.

Resolving patch:
diff --git a/ibis/backends/bigquery/__init__.py b/ibis/backends/bigquery/__init__.py
--- a/ibis/backends/bigquery/__init__.py
+++ b/ibis/backends/bigquery/__init__.py
@@ -81,7 +81,9 @@ def _create_client_info_gapic(application_name):
     return ClientInfo(user_agent=_create_user_agent(application_name))
 
 
-_MEMTABLE_PATTERN = re.compile(r"^_?ibis_(?:pandas|pyarrow)_memtable_[a-z0-9]{26}$")
+_MEMTABLE_PATTERN = re.compile(
+    r"^_?ibis_(?:[A-Za-z_][A-Za-z_0-9]*)_memtable_[a-z0-9]{26}$"
+)
 
 
 def _qualify_memtable(
@@ -433,9 +435,11 @@ def do_connect(
                 location=location,
             )
 
-        self.client.default_query_job_config = bq.QueryJobConfig(
-            use_legacy_sql=False, allow_large_results=True
-        )
+        if self.client.default_query_job_config is None:
+            self.client.default_query_job_config = bq.QueryJobConfig()
+
+        self.client.default_query_job_config.use_legacy_sql = False
+        self.client.default_query_job_config.allow_large_results = True
 
         if storage_client is not None:
             self.storage_client = storage_client


Test patch: