Repository: ibis-project/ibis

Issue:
CLN: Better database API
The behavior for the database API is not ideal in my opinion. For SQL databases, we can have something like:
```python
>>> con.list_databases()
['db1', 'db2', 'db3']
>>> con.current_database
'db2'
```
In #2909, #2910 and #2911 I'm making the API more consistent, like making sure that `current_database` always exist, or `list_databases` has always the same API. But the behavior is not consistent with the above. For example, the pandas backend is returning an empty list for the list of databases, and `None` for the current database. Or the file backends returning the list of directories in the path as databases, and `None` as the current database.

What I think would make things simpler, is that backends that can't have more than one database, use `main` as the current_database, and `list_databases()` returns `['main']`. This seems consistent with what SQLAlchemy does with SQLite (which doesn't support databases). And would make code simpler, always having database names, and having the current database name in the list of databases.

I'd also implement this for the file backends. What I would do is to just support one "database", the current directory (except for hdf5, which does support databases). Or if anyone thinks it's useful, we could find all directories with files of the backend format, and list those directories as the databases (can be in a follow up once it's needed).

Let me know if there are any objections and I'll implement it this way.


Discussion:
i am not sure this is a good change

multiple database are a thing and you are removing it for why?
> i am not sure this is a good change
> 
> multiple database are a thing and you are removing it for why?

For backends with multiple database support, such as postgres, mysql, impala, pyspark... I would keep things as they are. `list_databases()` returning all them, and `current_database` returning the default one.

This is for backends such as sqlite or pandas, where instead of not implementing the methods, or returning an empty list for `list_databases()` and `None` for `current_database`, I'd have a `main` database (or `default`, or whatever name), so the API is more intuitive.

Sorry I wasn't clear. Does this make sense?
I don't think it's a good idea to overlay an arbitrary notion of a database on top of things for which that doesn't make sense without more detail worked out.

I don't think it's more intuitive to get back `["main"]` for a collection of files in a directory, because `main` doesn't mean anything. It could mean something, but we can't introduce that without more specification of what it would mean for all the various non-database backends to have a database called main.

`None` makes sense, because it's the thing you typically use in Python to mean "nothing", and these backends have nothing to say when it comes to databases.

>  This seems consistent with what SQLAlchemy does with SQLite (which doesn't support databases)

The conclusion and the premise are false here: SQLite most definitely does support databases (https://www.sqlite.org/lang_attach.html, [we actually use this in ibis](https://github.com/ibis-project/ibis/blob/master/ibis/backends/sqlite/client.py#L52)), and the name `main` comes from SQLite, not SQLAlchemy:

```
❯ sqlite3 -column -header <<< 'pragma database_list'
seq  name  file
---  ----  ----
0    main
```

> we could find all directories with files of the backend format, and list those directories as the databases (can be in a follow up once it's needed).

This strikes me as an arbitrary choice that some people may or may not want made for them when accessing `.database`. IMO it's better to return `None` or even raise an exception to inform users that a database isn't a thing on CSV, Parquet, etc.

I am -1 on the specific change proposed here.

I am +1 on improving the database API in other ways though.
Just to make sure we're in the same page. There are three different cases here:
- Engines with database support (e.g. postgres)
- Backends with no database concept (e.g. pandas)
- File backends, where databases are mostly emulated with directories

I'm fine not adding arbitrary databases. But I don't think we should be using `None`, I think it makes things less intuitive in the user side. At least, not have this being returned:
```python
>>> con.current_database
None
>>> con.list_databases()
[]
```
I'm fine for pandas to raise a `NotImplementedError`. But the above makes things less intuitive IMO (like `len(con.list_databases())` returning `0`, or `con.current_database in con.list_databases()` returning `False`).

For the file backends, if we want to emulate databases with directories, I think `current_database` should be `.`. Would be interesting to have some real cases on when working with multiple directories is useful. Maybe using tables names like `my_dir/my_file.csv` could make things simpler, instead of emulating databases.
> I'm fine for pandas to raise a NotImplementedError. But the above makes things less intuitive IMO (like len(con.list_databases()) returning 0, or con.current_database in con.list_databases() returning False).

I'm not following what is less intuitive here. Can you clarify precisely what is not intuitive?

> For the file backends, if we want to emulate databases with directories, I think current_database should be ..

IMO it is questionable to continue supporting directories-as-databases.

A collection of files with a name _could_ be called a database, but I think something in between a database and a table (maybe a `Dataset`?) would be better suited to the "collection of files" case and lower expectations about what users should expect from different backends in ibis.

Personally, I think we should probably deprecate directories-as-files and raise an exception in future releases, followed by the design of an API that allows sane treatment of directories of files.

What do you think?
> I'm not following what is less intuitive here. Can you clarify precisely what is not intuitive?

Maybe it's just me. Imagine I'm using Ibis in postgres, I'm connected to a database, and I can see available databases using `list_databases()`. Then I transition to the pandas backend, and `list_databases()` returns `[]`, it feels like something is broken, and that something needs to be done or can be done, so that function returns a list of databases.

Also, I could get some code in a backend, where I  save the current database, and check in a different connection if it exists in the list of available databases. This would fail silently if current database is `None`, or if the list of databases is empty.

I understand returning `None` or an empty list is intuitive by itself in an interactive session. But I don't think it's ideal, in more complex cases. Not a big deal, but I would prefer raising an exception (or always having at least a database

).

> Personally, I think we should probably deprecate directories-as-files and raise an exception in future releases, followed by the design of an API that allows sane treatment of directories of files.

+1 to deprecate directories as databases. Or at least to improve the API. Having something like:
```python
>>> con.current_database
'.'
>>> con.list_databases()
['.', 'subdir1', 'subdir1/nested_subdir', 'subdir2']
>>> con.set_database('subdir2')
>>> con.current_database
'subdir2'
>>> con.list_databases()
['.', 'subdir1', 'subdir1/nested_subdir', 'subdir2']
```
would still make sense to me. I prefer to deprecate unless there is a good use case for it. But keeping what we have is very confusing IMHO.

Resolving patch:
diff --git a/ibis/backends/pandas/__init__.py b/ibis/backends/pandas/__init__.py
--- a/ibis/backends/pandas/__init__.py
+++ b/ibis/backends/pandas/__init__.py
@@ -64,10 +64,10 @@ def version(self) -> str:
 
     @property
     def current_database(self):
-        return 'main'
+        raise NotImplementedError('pandas backend does not support databases')
 
     def list_databases(self, like=None):
-        return self._filter_with_like(['main'])
+        raise NotImplementedError('pandas backend does not support databases')
 
     def list_tables(self, like=None, database=None):
         return self._filter_with_like(


Test patch:
diff --git a/ibis/backends/tests/conftest.py b/ibis/backends/tests/conftest.py
--- a/ibis/backends/tests/conftest.py
+++ b/ibis/backends/tests/conftest.py
@@ -285,11 +285,12 @@ def temp_view(con) -> str:
 @pytest.fixture(scope='session')
 def current_data_db(con, backend) -> str:
     """Return current database name."""
-    if not hasattr(con, 'current_database'):
+    try:
+        return con.current_database
+    except NotImplementedError:
         pytest.skip(
-            f'{backend.name()} backend doesn\'t have current_database method.'
+            f"{backend.name()} backend doesn't have current_database method."
         )
-    return con.current_database
 
 
 @pytest.fixture
diff --git a/ibis/backends/tests/test_api.py b/ibis/backends/tests/test_api.py
--- a/ibis/backends/tests/test_api.py
+++ b/ibis/backends/tests/test_api.py
@@ -1,3 +1,6 @@
+import pytest
+
+
 def test_backend_name(backend):
     # backend is the TestConf for the backend
     assert backend.api.name == backend.name()
@@ -7,6 +10,7 @@ def test_version(backend):
     assert isinstance(backend.api.version, str)
 
 
+@pytest.mark.xfail_unsupported
 def test_database_consistency(con):
     # every backend has a different set of databases, not testing the
     # exact names for now