Repository: ibis-project/ibis

Issue:
bug: RisingWave race condition (?) where table returns before data is loaded
### What happened?

wanted to show off using RisingWave, and first needed to load in some data. separate issue that `to_pyarrow()` doesn't work in place of `to_pandas()` here, but that's also an issue in Postgres

surprisingly though, when the table is created it is first empty -- after some time (probably less than a second in this case) the data seems to be populated

after `just up risingwave`:

```python
[ins] In [1]: import ibis

[ins] In [2]: ibis.options.interactive = True

[ins] In [3]: con = ibis.risingwave.connect(host="localhost", port=4566, database="dev", user="postgres")

[ins] In [4]: t = ibis.examples.penguins.fetch()

[nav] In [5]: t = con.create_table("penguins", t.to_pandas(), overwrite=True)

[ins] In [6]: t
Out[6]:
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━┓
┃ species              ┃ island               ┃ bill_length_mm       ┃ bill_depth_mm        ┃ flipper_length_mm    ┃ body_mass_g          ┃ … ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━┩
│ string               │ string               │ float64              │ float64              │ float64              │ float64              │ … │
└──────────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴───┘

[ins] In [7]: t
Out[7]:
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━┓
┃ species              ┃ island               ┃ bill_length_mm       ┃ bill_depth_mm        ┃ flipper_length_mm    ┃ body_mass_g          ┃ … ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━┩
│ string               │ string               │ float64              │ float64              │ float64              │ float64              │ … │
└──────────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴───┘

[ins] In [8]: t
Out[8]:
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ species   ┃ island    ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ string    │ string    │ float64        │ float64       │ float64           │ float64     │ string │ int64 │
├───────────┼───────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ Gentoo    │ Biscoe    │           50.5 │          15.9 │             225.0 │      5400.0 │ male   │  2008 │
│ Chinstrap │ Dream     │           52.2 │          18.8 │             197.0 │      3450.0 │ male   │  2009 │
│ Adelie    │ Dream     │           33.1 │          16.1 │             178.0 │      2900.0 │ female │  2008 │
│ Adelie    │ Torgersen │           40.6 │          19.0 │             199.0 │      4000.0 │ male   │  2009 │
│ Gentoo    │ Biscoe    │           50.1 │          15.0 │             225.0 │      5000.0 │ male   │  2008 │
│ Gentoo    │ Biscoe    │           48.7 │          14.1 │             210.0 │      4450.0 │ female │  2007 │
│ Gentoo    │ Biscoe    │           45.0 │          15.4 │             220.0 │      5050.0 │ male   │  2008 │
│ Gentoo    │ Biscoe    │           44.9 │          13.8 │             212.0 │      4750.0 │ female │  2009 │
│ Adelie    │ Dream     │           37.5 │          18.5 │             199.0 │      4475.0 │ male   │  2009 │
│ Adelie    │ Dream     │           38.1 │          17.6 │             187.0 │      3425.0 │ female │  2009 │
│ …         │ …         │              … │             … │                 … │           … │ …      │     … │
└───────────┴───────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘
```

### What version of ibis are you using?

main

### What backend(s) are you using, if any?

RisingWave

### Relevant log output

```sh
n/a
```


### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
So either by design or by accident, `psycopg2` is returning before the entire query has executed.  Given that this doesn't happen with postgres and we're using the same logic in `create_table`, I think this is likely an upstream issue (or possibly a feature)
@KeXiangWang any thoughts on this? 
I tried this out on our 8.0 release and it doesn't have this issue, so something in between Ibis and Risingwave is acting a little funky.

I tried explicitly enabling `autocommit` but it didn't help.
Thanks for mentioning me. I'll take a look tmr. 
It's actually an expected behavior. Due to some unique version control in Risingwave. The data recently written to RW will be available after 1 or 2 second. Usually we recommend users to use `FLUSH` to wait until the written data is available or use `SET RW_IMPLICIT_FLUSH TO true;` to set the session to be flushed automatically. The detailed explanation can be found [here](https://docs.risingwave.com/docs/current/sql-flush/) and [here](https://docs.risingwave.com/docs/current/view-configure-runtime-parameters/#how-to-view-runtime-parameters:~:text=Description-,rw_implicit_flush,-true/false).

Resolving patch:
diff --git a/ibis/backends/risingwave/__init__.py b/ibis/backends/risingwave/__init__.py
--- a/ibis/backends/risingwave/__init__.py
+++ b/ibis/backends/risingwave/__init__.py
@@ -119,6 +119,7 @@ def do_connect(
 
         with self.begin() as cur:
             cur.execute("SET TIMEZONE = UTC")
+            cur.execute("SET RW_IMPLICIT_FLUSH TO true;")
 
     def create_table(
         self,


Test patch: