internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ibis-project/ibis

Issue:
bug(duckdb): handle `TIMESTAMP_NS` parsing



Discussion:
Can we handle any arbitrary types that we don't know about already, by just returning some junk Null type?  That way existing code won't be broken by adding a new column with an unknown type to a table.

Resolving patch:
diff --git a/ibis/backends/duckdb/datatypes.py b/ibis/backends/duckdb/datatypes.py
--- a/ibis/backends/duckdb/datatypes.py
+++ b/ibis/backends/duckdb/datatypes.py
@@ -36,6 +36,7 @@
     int16,
     int32,
     int64,
+    json,
     spaceless,
     spaceless_string,
     string,
@@ -63,9 +64,14 @@ def parse(text: str, default_decimal_parameters=(18, 3)) -> DataType:
         | spaceless_string("double", "float8").result(float64)
         | spaceless_string("real", "float4", "float").result(float32)
         | spaceless_string("smallint", "int2", "short").result(int16)
-        | spaceless_string("timestamp", "datetime").result(
-            Timestamp(timezone="UTC")
-        )
+        | spaceless_string(
+            "timestamp_tz",
+            "timestamp_sec",
+            "timestamp_ms",
+            "timestamp_ns",
+            "timestamp",
+            "datetime",
+        ).result(Timestamp(timezone="UTC"))
         | spaceless_string("date").result(date)
         | spaceless_string("time").result(time)
         | spaceless_string("tinyint", "int1").result(int8)
@@ -82,6 +88,7 @@ def parse(text: str, default_decimal_parameters=(18, 3)) -> DataType:
             "text",
             "string",
         ).result(string)
+        | spaceless_string("json").result(json)
     )
 
     @p.generate


Test patch:
diff --git a/ibis/backends/duckdb/tests/test_datatypes.py b/ibis/backends/duckdb/tests/test_datatypes.py
--- a/ibis/backends/duckdb/tests/test_datatypes.py
+++ b/ibis/backends/duckdb/tests/test_datatypes.py
@@ -58,6 +58,11 @@
     ),
     T=dt.Array(dt.Array(dt.int32)),
     U=dt.Array(dt.Array(dt.int32)),
+    V=dt.Timestamp("UTC"),
+    W=dt.Timestamp("UTC"),
+    X=dt.Timestamp("UTC"),
+    Y=dt.Timestamp("UTC"),
+    Z=dt.json,
 )
 
 
@@ -113,6 +118,11 @@
             ("S", "STRUCT(a INT, b TEXT, c LIST<MAP<TEXT, LIST<FLOAT8>>>)"),
             ("T", "LIST<LIST<INTEGER>>"),
             ("U", "INTEGER[][]"),
+            ("V", "TIMESTAMP_TZ"),
+            ("W", "TIMESTAMP_SEC"),
+            ("X", "TIMESTAMP_MS"),
+            ("Y", "TIMESTAMP_NS"),
+            ("Z", "JSON"),
         ]
     ],
 )
Repository: ibis-project/ibis

Issue:
bug: `ibis.connect` not registering csv files correctly
As noted in #4542 , the pattern `con = ibis.connect('./local.csv')` doesn't do the expected thing, which would be to throw it into `duckdb` and then return a connection, but it is nominally supported in the `connect` dispatcher.  


Discussion:
Oh, interesting! Well, in that case there's not much to do except make it work :)
blarg, looks like a priority battle in the regex dispatcher inside of `connect` -- easy to make CSVs load but then other functionality breaks

Resolving patch:
diff --git a/ibis/backends/base/__init__.py b/ibis/backends/base/__init__.py
--- a/ibis/backends/base/__init__.py
+++ b/ibis/backends/base/__init__.py
@@ -760,4 +760,6 @@ def _(filename: str, **kwargs: Any) -> BaseBackend:
     >>> con = ibis.connect("relative/path/to/data.csv")
     >>> con = ibis.connect("relative/path/to/more/data.parquet")
     """
-    return _connect(f"duckdb:///{filename}", **kwargs)
+    con = ibis.duckdb.connect()
+    con.register(filename)
+    return con


Test patch:
diff --git a/ibis/backends/tests/test_client.py b/ibis/backends/tests/test_client.py
--- a/ibis/backends/tests/test_client.py
+++ b/ibis/backends/tests/test_client.py
@@ -497,6 +497,22 @@ def test_connect_file_url(url, tmp_db):
     assert con.execute(one) == 1
 
 
+@pytest.mark.duckdb
+@pytest.mark.parametrize(
+    "out_method, extension",
+    [
+        ("to_csv", "csv"),
+        ("to_parquet", "parquet"),
+    ],
+)
+def test_connect_local_file(
+    out_method, extension, test_employee_data_1, tmp_path
+):
+    getattr(test_employee_data_1, out_method)(tmp_path / f"out.{extension}")
+    con = ibis.connect(tmp_path / f"out.{extension}")
+    assert con.list_tables()
+
+
 @pytest.mark.duckdb
 @pytest.mark.parametrize(
     "read_only",
Repository: ibis-project/ibis

Issue:
bug: `.visualize(label_edges=True)` case ops.NodeList we get ValueError tuple.index(x): x not in tuple
Hi,

There's still one small thing that needs fixing in the great `.visualize(label_edges=True)` feature 😄 :

When running...
```py
import ibis
t = ibis.table((("a", "int32"), ("b", "string")))
expr = t[(t["a"] == 1) & (t["b"] == "x")]

expr.visualize(label_edges=True)
```
...I get:
```
Exception has occurred: ValueError - tuple.index(x): x not in tuple
```
at the following line:
https://github.com/ibis-project/ibis/blob/2c9cfea15fc4d5f61e9099c3b270ea61498b5e45/ibis/expr/visualize.py#L117

This is happening when `v` is an `ops.NodeList` and thus its `v.args` is a tuple `of tuples` and NOT a tuple of nodes.

Given that on the next line we have special logic for `ops.NodeList`, maybe one quick fix could be to use the `.values`/`.args[0]` in such cases:

![image](https://user-images.githubusercontent.com/10154357/192848219-48560533-9cbc-4621-8759-145dcbe95462.png)

Thank you.


Discussion:

Resolving patch:
diff --git a/ibis/expr/visualize.py b/ibis/expr/visualize.py
--- a/ibis/expr/visualize.py
+++ b/ibis/expr/visualize.py
@@ -114,10 +114,11 @@ def to_graph(expr, node_attr=None, edge_attr=None, label_edges: bool = False):
                 if not label_edges:
                     label = None
                 else:
-                    index = v.args.index(u)
                     if isinstance(v, ops.NodeList):
+                        index = v.values.index(u)
                         arg_name = f"values[{index}]"
                     else:
+                        index = v.args.index(u)
                         arg_name = v.argnames[index]
                     label = f"<.{arg_name}>"
 


Test patch:
diff --git a/ibis/tests/expr/test_visualize.py b/ibis/tests/expr/test_visualize.py
--- a/ibis/tests/expr/test_visualize.py
+++ b/ibis/tests/expr/test_visualize.py
@@ -152,6 +152,15 @@ def test_asof_join():
     assert key(result.op()) in graph.source
 
 
+def test_filter():
+    # Smoketest that NodeList nodes are handled properly
+    t = ibis.table([('a', 'int64'), ('b', 'string'), ('c', 'int32')])
+    expr = t.filter((t.a == 1) & (t.b == "x"))
+    graph = viz.to_graph(expr, label_edges=True)
+    assert "values[0]" in graph.source
+    assert "values[1]" in graph.source
+
+
 def test_html_escape(with_graphviz):
     # Check that we correctly escape HTML <> characters in the graphviz
     # representation. If an error is thrown, _repr_png_ returns None.
Repository: ibis-project/ibis

Issue:
Problem with formatting union expressions when using `value_counts`
I'm working on a subclass of the MySQL backend and using unions. When attempting to do a `value_counts` on a union, I get an attribute error. Here is a simple test using our backend (this DataFrame upload might not work in the actual MySQL, but should be fairly simple to replicate).
```
df = pd.DataFrame([[0, 1], [2, 3]], columns=['a', 'b'])
tbl = conn.create_table('test_union', df, force=True)
u = ibis.union(tbl, tbl)
u.a.value_counts()
```

Here is the tail end of the exception.
```
...
~/.pyenv/versions/3.9.4/lib/python3.9/site-packages/ibis/expr/format.py in _fmt_value_expr(expr, aliases)
    555     Forwards the call on to the specific operation dispatch rule.
    556     """
--> 557     return fmt_value(expr.op(), aliases=aliases)
    558 
    559 

~/.pyenv/versions/3.9.4/lib/python3.9/functools.py in wrapper(*args, **kw)
    875                             '1 positional argument')
    876 
--> 877         return dispatch(args[0].__class__)(*args, **kw)
    878 
    879     funcname = getattr(func, '__name__', 'singledispatch function')

~/.pyenv/versions/3.9.4/lib/python3.9/site-packages/ibis/expr/format.py in _fmt_value_table_node(op, aliases, **_)
    669     if not hasattr(op, 'table'):
    670         import pdb; pdb.set_trace()
--> 671     return f"{aliases[op.table.op()]}"
    672 
    673 

AttributeError: 'Union' object has no attribute 'table'
```


Discussion:
Thanks for opening this, this is a bug in the expr repr (the error comes from repring `u.a.value_counts()`, not the generation of the expression). If you avoid ever repr'ing the expression, or enable interactive mode

```
ibis.options.interactive = True
```

you should be able to work around this for now.

Should be an easy fix though, I can look into this later.

Resolving patch:
diff --git a/ibis/expr/format.py b/ibis/expr/format.py
--- a/ibis/expr/format.py
+++ b/ibis/expr/format.py
@@ -643,7 +643,7 @@ def _fmt_value_table_node(op: ops.TableNode, *, aliases: Aliases, **_: Any) -> s
     This function is called when a table is used in a value expression.
     An example is `table.count()`.
     """
-    return f"{aliases[op.table]}"
+    return f"{aliases[op]}"
 
 
 @fmt_value.register


Test patch:
diff --git a/ibis/tests/expr/test_format.py b/ibis/tests/expr/test_format.py
--- a/ibis/tests/expr/test_format.py
+++ b/ibis/tests/expr/test_format.py
@@ -275,6 +275,21 @@ def test_tables_have_format_value_rules(cls):
     assert cls in ibis.expr.format.fmt_value.registry
 
 
+@pytest.mark.parametrize(
+    "f",
+    [
+        lambda t1, t2: t1.count(),
+        lambda t1, t2: t1.join(t2, t1.a == t2.a).count(),
+        lambda t1, t2: ibis.union(t1, t2).count(),
+    ],
+)
+def test_table_value_expr(f):
+    t1 = ibis.table([("a", "int"), ("b", "float")], name="t1")
+    t2 = ibis.table([("a", "int"), ("b", "float")], name="t2")
+    expr = f(t1, t2)
+    repr(expr)  # smoketest
+
+
 def test_window_no_group_by():
     t = ibis.table(dict(a="int64", b="string"), name="t")
     expr = t.a.mean().over(ibis.window(preceding=0))
Repository: ibis-project/ibis

Issue:
refactor: remove `sorted` calls from mutation
        I thought we had removed `sorted`, but I guess not. We should probably do that for 4.0.0.

_Originally posted by @cpcloud in https://github.com/ibis-project/ibis/pull/4649#discussion_r996311096_

Let's remove `sorted` from mutation calls since `dict`s are now ordered.


Discussion:

Resolving patch:
diff --git a/ibis/expr/types/relations.py b/ibis/expr/types/relations.py
--- a/ibis/expr/types/relations.py
+++ b/ibis/expr/types/relations.py
@@ -601,9 +601,7 @@ def ensure_expr(expr):
 
         exprs = [] if exprs is None else util.promote_list(exprs)
         exprs = [ensure_expr(expr) for expr in exprs]
-        exprs.extend(
-            ensure_expr(expr).name(name) for name, expr in sorted(mutations.items())
-        )
+        exprs.extend(ensure_expr(expr).name(name) for name, expr in mutations.items())
         mutation_exprs = an.get_mutation_exprs(exprs, self)
         return self.select(mutation_exprs)
 


Test patch:
Repository: ibis-project/ibis

Issue:
feat(pandas): support `substr` with `length=None`
The `substr` function provides a default value for the `length` parameter (`None`), and [the docs](https://ibis-project.org/docs/3.0.2/api/expressions/strings/#ibis.expr.types.strings.StringValue.substr) say `length` is optional

However the Pandas backend doesn't have an implementation for `substr` without `length`

```
import pandas as pd

import ibis

backend = ibis.backends.pandas.Backend()
conn = backend.connect({})

table = conn.from_dataframe(pd.DataFrame({
    'key': [1, 2, 3],
    'value': ['foo', 'bar', 'baz'],
}), 't1')

table = table.mutate(
    value2=table['value'].substr(2)
)

table.execute()
```

```
NotImplementedError: Could not find signature for execute_node: <Substring, Series, int, NoneType>
```

I was only able to test this on `3.0.0`, but I did a quick look at [the latest Pandas backend code for `Substring`](https://github.com/ibis-project/ibis/blob/993400c35d8f87d1c0a7c22f1355e6dde885f1cf/ibis/backends/pandas/execution/strings.py#L23-L55) and I think the behavior is the same


Discussion:

Resolving patch:
diff --git a/ibis/backends/dask/execution/strings.py b/ibis/backends/dask/execution/strings.py
--- a/ibis/backends/dask/execution/strings.py
+++ b/ibis/backends/dask/execution/strings.py
@@ -54,7 +54,7 @@
             (
                 dd.Series,
                 integer_types,
-                integer_types,
+                (type(None), *integer_types),
             ),
             execute_substring_int_int,
         ),
diff --git a/ibis/backends/pandas/execution/strings.py b/ibis/backends/pandas/execution/strings.py
--- a/ibis/backends/pandas/execution/strings.py
+++ b/ibis/backends/pandas/execution/strings.py
@@ -21,9 +21,14 @@ def execute_string_length_series(op, data, **kwargs):
     return data.str.len().astype('int32')
 
 
-@execute_node.register(ops.Substring, pd.Series, integer_types, integer_types)
+@execute_node.register(
+    ops.Substring, pd.Series, integer_types, (type(None), *integer_types)
+)
 def execute_substring_int_int(op, data, start, length, **kwargs):
-    return data.str[start : start + length]
+    if length is None:
+        return data.str[start:]
+    else:
+        return data.str[start : start + length]
 
 
 @execute_node.register(ops.Substring, pd.Series, pd.Series, integer_types)


Test patch:
diff --git a/ibis/backends/tests/test_string.py b/ibis/backends/tests/test_string.py
--- a/ibis/backends/tests/test_string.py
+++ b/ibis/backends/tests/test_string.py
@@ -204,6 +204,12 @@ def test_string_col_is_unicode(alltypes, df):
             lambda t: t.date_string_col.str[2:5],
             id='substr',
         ),
+        param(
+            lambda t: t.date_string_col.substr(2),
+            lambda t: t.date_string_col.str[2:],
+            id='substr-start-only',
+            marks=pytest.mark.notimpl(["datafusion", "polars"]),
+        ),
         param(
             lambda t: t.date_string_col.left(2),
             lambda t: t.date_string_col.str[:2],
Repository: ibis-project/ibis

Issue:
docs: infinite build when using `mkdocs serve`
It appears that when using `mkdocs serve` the docs are repeatedly rebuilt to no end.

I suspect there's a file that we're generating (maybe the operation matrix?) that is being considered new and triggering a rebuild.


Discussion:

Resolving patch:
diff --git a/gen_matrix.py b/gen_matrix.py
--- a/gen_matrix.py
+++ b/gen_matrix.py
@@ -69,7 +69,15 @@ def main():
         "backends",
         "support_matrix.csv",
     )
-    table.to_csv(dst, index_label="Backends")
+
+    if dst.exists():
+        old = pd.read_csv(dst, index_col="Backends")
+        should_write = not old.equals(table)
+    else:
+        should_write = True
+
+    if should_write:
+        table.to_csv(dst, index_label="Backends")
 
 
 main()


Test patch:
Repository: ibis-project/ibis

Issue:
bug: visualize label_edges fails after recent (?) merged PRs
Hey,

Just synced my local repo with ibis master and I noticed:

- a GOOD thing:

   The `predicates` of `ops.Join`(s) are now returned by the `Graph.from_bfs`  -> in recent earlier versions they were NOT returned (I wanted to open an issue about that, but didn't want to bother you with non-essential feature requests)

- an...OTHER thing: 😄 
  Because these `ops.Join predicates` recently appeared (?), the following snippet now started failing:

  ```py
  import ibis
  
  t = ibis.table((("a", "int32"), ("b", "string")))
  t1 = ibis.table((("c", "int32"), ("b", "string")))
  
  expr = t[(t["a"] == 1) & (t["b"] == "x")]
  joined = expr.join(t1, (t["b"] == t1["b"]))
  
  joined.visualize(label_edges=True)
  ```
  - The problem:
    ![image](https://user-images.githubusercontent.com/10154357/194530692-9fce060d-af78-41d6-b6b7-2a3841f07df9.png)

  - A quick-and-really-dirty fix:

![image](https://user-images.githubusercontent.com/10154357/194578747-cd181217-60ce-43a3-ae84-3cb6e9cf7c28.png)

  - A question - **pls don't answer if it takes you a long time**:
    **For me it seems wierd that the `ops.Join predicates` are a `tuple` (!) and not a `NodeList` like the predicates of other nodes. Why aren't they a `NodeList`?**

After fixing the bug, the `ops.Join predicates` are shown as expected:

![image](https://user-images.githubusercontent.com/10154357/194532833-eae82596-48fa-4543-8545-4eb47d6b7de4.png)

Thanks



Discussion:

Resolving patch:
diff --git a/ibis/expr/visualize.py b/ibis/expr/visualize.py
--- a/ibis/expr/visualize.py
+++ b/ibis/expr/visualize.py
@@ -115,6 +115,10 @@ def to_graph(expr, node_attr=None, edge_attr=None, label_edges: bool = False):
                     if isinstance(v, ops.NodeList):
                         index = v.values.index(u)
                         arg_name = f"values[{index}]"
+                    elif isinstance(v, ops.Join):
+                        arg_name = ""
+                        if u in v.predicates:
+                            arg_name = "predicates"
                     else:
                         index = v.args.index(u)
                         arg_name = v.argnames[index]


Test patch:
Repository: ibis-project/ibis

Issue:
bug: S3 Absolute Paths are Not Casted as pyarrow datasets
### What happened?

When trying to directly read a .parquet file in S3 I was getting a weird duckdb error...after a lot of testing I was able to determine that the `_generate_view_code.register` for full s3 paths to parquet files was not pointing to the S3 function for that _generate_view_code but instead the parquet file _generate_view_code.

Example:

```py
import ibis


from ibis.backends.duckdb import _generate_view_code


s3_directory = "s3://data-lake/dataset/"  # contains many .parquet files...
s3_path = "s3://data-lake/dataset/file_1.parquet"

try:
    print(_generate_view_code(s3_directory))
# behaves normally as the above paths are not real
except OSError:
    pass

# below outputs a local path to a file instead of casting as pyarrow.dataset like the above
print(_generate_view_code(s3_path))
"""
('CREATE OR REPLACE VIEW "/Users/jslesar/beta/ibis/s3:/data-lake/dataset/file_1_parquet" AS\nSELECT * FROM read_parquet(\'/Users/jslesar/beta/ibis/s3:/data-lake/dataset/file_1.parquet\')', '/Users/jslesar/beta/ibis/s3:/data-lake/dataset/file_1_parquet', [])
"""
```
 

### What version of ibis are you using?

Latest master

### What backend(s) are you using, if any?

duckdb

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
bumping priority to 13 locally seemed to fix https://github.com/ibis-project/ibis/blob/master/ibis/backends/duckdb/__init__.py#L112

Going to submit an MR in a bit

Resolving patch:
diff --git a/ibis/backends/duckdb/__init__.py b/ibis/backends/duckdb/__init__.py
--- a/ibis/backends/duckdb/__init__.py
+++ b/ibis/backends/duckdb/__init__.py
@@ -109,7 +109,7 @@ def _parquet_file_or_url(_, path, table_name=None, **kwargs):
     return _parquet(f"parquet://{path}", path=path, table_name=table_name, **kwargs)
 
 
-@_generate_view_code.register(r"s3://.+", priority=10)
+@_generate_view_code.register(r"s3://.+", priority=13)
 def _s3(full_path, table_name=None):
     # TODO: gate this import once the ResultHandler mixin is merged #4454
     import pyarrow.dataset as ds


Test patch:
diff --git a/ibis/backends/duckdb/tests/test_register.py b/ibis/backends/duckdb/tests/test_register.py
--- a/ibis/backends/duckdb/tests/test_register.py
+++ b/ibis/backends/duckdb/tests/test_register.py
@@ -213,3 +213,11 @@ def test_temp_directory(tmp_path):
     con = ibis.duckdb.connect(tmp_path / "test2.ddb", temp_directory=temp_directory)
     [(value,)] = con.con.execute(query).fetchall()
     assert value == str(temp_directory)
+
+
+@pytest.mark.parametrize(
+    "path", ["s3://data-lake/dataset/", "s3://data-lake/dataset/file_1.parquet"]
+)
+def test_s3_parquet(path):
+    with pytest.raises(OSError):
+        _generate_view_code(path)
Repository: ibis-project/ibis

Issue:
bug(snowflake): snowflake doesn't like Tuesdays
### What happened?

```
In [1]: import ibis, os; ibis.options.interactive = True; from ibis import _

In [2]: con = ibis.connect(os.environ["SNOWFLAKE_URL"])

In [3]: con.execute(ibis.literal('2017-01-10').cast('date').day_of_week.full_name())

In [4]: con.execute(ibis.literal('2017-01-17').cast('date').day_of_week.full_name())

In [5]: con.execute(ibis.literal('2017-01-24').cast('date').day_of_week.full_name())

In [6]: con = ibis.duckdb.connect()

In [7]: con.execute(ibis.literal('2017-01-24').cast('date').day_of_week.full_name())
Out[7]: 'Tuesday'

In [8]: con = ibis.connect(os.environ["SNOWFLAKE_URL"])

In [9]: con.execute(ibis.literal('2017-01-23').cast('date').day_of_week.full_name())
Out[9]: 'Monday'

In [10]: con.execute(ibis.literal('2017-01-25').cast('date').day_of_week.full_name())
Out[10]: 'Wednesday'
```

I expected Snowflake to be fine with Tuesday.

### What version of ibis are you using?

`master`

### What backend(s) are you using, if any?

Snowflake, DuckDB

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:

Resolving patch:
diff --git a/ibis/backends/snowflake/registry.py b/ibis/backends/snowflake/registry.py
--- a/ibis/backends/snowflake/registry.py
+++ b/ibis/backends/snowflake/registry.py
@@ -47,7 +47,7 @@ def _day_of_week_name(t, op):
         whens=[
             ("Sun", "Sunday"),
             ("Mon", "Monday"),
-            ("Tues", "Tuesday"),
+            ("Tue", "Tuesday"),
             ("Wed", "Wednesday"),
             ("Thu", "Thursday"),
             ("Fri", "Friday"),


Test patch:
diff --git a/ibis/backends/tests/test_temporal.py b/ibis/backends/tests/test_temporal.py
--- a/ibis/backends/tests/test_temporal.py
+++ b/ibis/backends/tests/test_temporal.py
@@ -710,17 +710,7 @@ def test_string_to_timestamp_tz_error(alltypes):
     [
         param('2017-01-01', 6, 'Sunday', id="sunday"),
         param('2017-01-02', 0, 'Monday', id="monday"),
-        param(
-            '2017-01-03',
-            1,
-            'Tuesday',
-            id="tuesday",
-            marks=[
-                pytest.mark.notyet(
-                    ["snowflake"], reason="snowflake is not a fan of Tuesdays"
-                )
-            ],
-        ),
+        param('2017-01-03', 1, 'Tuesday', id="tuesday"),
         param('2017-01-04', 2, 'Wednesday', id="wednesday"),
         param('2017-01-05', 3, 'Thursday', id="thursday"),
         param('2017-01-06', 4, 'Friday', id="friday"),