Repository: ibis-project/ibis

Issue:
feat: supprt empty ibis.spark.connect()
### Is your feature request related to a problem?

you can do:

```python
import ibis

con = ibis.connect("pyspark://")
```

(as long as you have PySpark and whatnot installed)

but not:

```python
import ibis

con = ibis.pyspark.connect()
```

you need to pass in the session object in the latter case; in the former, it seems to be automagically created on your behalf

of course, feel free to close if this is intentional or technical reasons for this behavior or whanot 

```python
[ins] In [1]: import ibis
c
[ins] In [2]: con = ibis.connect("pyspark://")
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/02/28 19:07:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/02/28 19:07:34 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
c
[ins] In [3]: con = ibis.pyspark.connect()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 con = ibis.pyspark.connect()

File ~/repos/ibis/ibis/__init__.py:107, in __getattr__.<locals>.connect(*args, **kwargs)
    106 def connect(*args, **kwargs):
--> 107     return backend.connect(*args, **kwargs)

File ~/repos/ibis/ibis/backends/__init__.py:764, in BaseBackend.connect(self, *args, **kwargs)
    740 """Connect to the database.
    741
    742 Parameters
   (...)
    761
    762 """
    763 new_backend = self.__class__(*args, **kwargs)
--> 764 new_backend.reconnect()
    765 return new_backend

File ~/repos/ibis/ibis/backends/__init__.py:778, in BaseBackend.reconnect(self)
    776 def reconnect(self) -> None:
    777     """Reconnect to the database already configured with connect."""
--> 778     self.do_connect(*self._con_args, **self._con_kwargs)

TypeError: Backend.do_connect() missing 1 required positional argument: 'session'
```

### Describe the solution you'd like

the `ibis.pyspark.connect()` just works

### What version of ibis are you running?

main

### What backend(s) are you using, if any?

pyspark

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:

Resolving patch:
diff --git a/ibis/backends/pyspark/__init__.py b/ibis/backends/pyspark/__init__.py
--- a/ibis/backends/pyspark/__init__.py
+++ b/ibis/backends/pyspark/__init__.py
@@ -130,7 +130,7 @@ def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self._cached_dataframes = {}
 
-    def do_connect(self, session: SparkSession) -> None:
+    def do_connect(self, session: SparkSession | None = None) -> None:
         """Create a PySpark `Backend` for use with Ibis.
 
         Parameters
@@ -147,6 +147,11 @@ def do_connect(self, session: SparkSession) -> None:
         <ibis.backends.pyspark.Backend at 0x...>
 
         """
+        if session is None:
+            from pyspark.sql import SparkSession
+
+            session = SparkSession.builder.getOrCreate()
+
         self._context = session.sparkContext
         self._session = session
 


Test patch:
diff --git a/ibis/backends/pyspark/tests/test_basic.py b/ibis/backends/pyspark/tests/test_basic.py
--- a/ibis/backends/pyspark/tests/test_basic.py
+++ b/ibis/backends/pyspark/tests/test_basic.py
@@ -125,3 +125,9 @@ def test_string_literal_backslash_escaping(con):
     expr = ibis.literal("\\d\\e")
     result = con.execute(expr)
     assert result == "\\d\\e"
+
+
+def test_connect_without_explicit_session():
+    con = ibis.pyspark.connect()
+    result = con.sql("SELECT CAST(1 AS BIGINT) as foo").to_pandas()
+    tm.assert_frame_equal(result, pd.DataFrame({"foo": [1]}))