Repository: ibis-project/ibis

Issue:
docs: refactor and move to Quarto
subsumes #6512 
related https://github.com/ibis-project/ibis/discussions/3701
related #6511 
related #6618 

---

The purpose of this issue is to propose and track the implementation of Ibis documentation in [Quarto](https://quarto.org).

There are several motivations for this work. The first is the existing documentation using mkdocs is slow to build and has a lot of minor issues. The most important reason for moving to Quarto is to improve the user experience of using Ibis documentation and key features like more native interactive Python support (in markdown files) offers both a better user and documentation developer experience. Additionally, numerous minor issues are fixed. 

## Timeline

We should aim to have fully moved our documentation over by the release of v7, but as soon as possible.

## Priorities

### P0

We must have these to consider switching to Quarto as main site.

- [x] CI/CD for new docs process (in progress https://github.com/ibis-project/ibis/pull/6937)
- [x] home page finalized
- [x] fix broken links (so many) (in progress https://github.com/ibis-project/ibis/pull/6937)
- [x] "Why use Ibis" article
- [x] "What is Ibis" article
- [x] installation page fixed
- [x] backend pages fixed
- [x] dplyr tutorial Quarto-fied
- [x] how-to guides structure finalized
- [x] how-to guides all Quarto-fied
- [x] reference documentation finished
- [x] release notes process updated (in progress https://github.com/ibis-project/ibis/pull/6937)
- [x] community documentation fixed
- [x] AI tab removed or updated (try it out!)

### P1

We should have these before merging.

- [ ] add metadata to all pages for SEO
- [ ] preview for docs changes on PR setup
- [ ] theme and font finalized
- [ ] redesigned logo (for new theme)
- [ ] quartodoc reference tool upstreamed (cc: @machow)
- [ ] community section updates (e.g. new async tool? documentation?)

### P2

Nice to have before merge; likely follow ups.

- [ ] backlog of tips to post
- [ ] new concepts
- [ ] pyspark tutorial
- [x] visualization tutorial (plotting libraries -- one or multiple)
- [x] new how-to guides
- [x] old posts Quarto-fied


Questionable items:

- [ ] automate unique image generation for posts?
- [ ] automate posts to social media?

## Organization, structure, opinions

Open to feedback.

I have started porting the docs to Quarto here: https://github.com/lostmygithubaccount/ibis/tree/quarto-docs.

See the [README in the docs2 directory](https://github.com/lostmygithubaccount/ibis/tree/quarto-docs/docs2#readme) for how to setup and preview the docs locally. The website has 5 primary sections mostly aligning with the https://diataxis.fr/ framework. Backends are important enough for their own section. Then, we follow the standard getting-started/concepts/how-to/reference split for the rest of the documentation.

The concepts are very light right now. It would be helpful to decide on the top-level structure here before writing all of the documents. I will take on drafting the "Why use Ibis" document based on recent discussions. 

The how-to guides are also light and again I think we should decide on a top-level structure. These should be search engine optimized and contain useful code snippets a user is likely to want. We should section them by task, e.g. "file input and output", "eda", "ml", "timeseries". I don't love the current layout though. 

We should limit backend-specific documentation to that section, i.e. disallow backend-specific how-to guides, to avoid sprawl.

All documentation should be written in `.qmd` files; we will convert old documents and if a user insists on PR'ing a notebook or otherwise we'll convert it over (this is easy and automatable). This format allows for significantly easier CI/CD to test examples (something both GCP and Azure have custom solutions in) by embedding the code directly in markdown. Modern IDE tooling (vim and VSCode, at least) make working with these files in what I would argue is a better experience than traditional notebooks. So, we will embrace this format for documentation to both improve the end user experience (less broken examples, better look) and developer experience (easier PRs). 

More thoughts on AI in a separate issue. Try out the app though, it seems decent especially for getting started. 


Discussion:

Resolving patch:
diff --git a/docs2/backends/app/backend_info_app.py b/docs2/backends/app/backend_info_app.py
new file mode 100644
--- /dev/null
+++ b/docs2/backends/app/backend_info_app.py
@@ -0,0 +1,207 @@
+from __future__ import annotations
+
+import datetime
+import tempfile
+from pathlib import Path
+from typing import Optional
+
+import pandas as pd
+import requests
+import sqlglot
+import streamlit as st
+
+import ibis
+from ibis import _
+
+ONE_HOUR_IN_SECONDS = datetime.timedelta(hours=1).total_seconds()
+
+st.set_page_config(layout="wide")
+
+# Track all queries. We display them at the bottom of the page.
+ibis.options.verbose = True
+sql_queries = []
+ibis.options.verbose_log = lambda sql: sql_queries.append(sql)
+
+
+@st.cache_data(ttl=ONE_HOUR_IN_SECONDS)
+def support_matrix_df():
+    resp = requests.get("https://ibis-project.org/backends/raw_support_matrix.csv")
+    resp.raise_for_status()
+
+    with tempfile.NamedTemporaryFile() as f:
+        f.write(resp.content)
+        return (
+            ibis.read_csv(f.name)
+            .relabel({"FullOperation": "full_operation"})
+            .mutate(
+                short_operation=_.full_operation.split(".")[-1],
+                operation_category=_.full_operation.split(".")[-2],
+            )
+            .to_pandas()
+        )
+
+
+@st.cache_data(ttl=ONE_HOUR_IN_SECONDS)
+def backends_info_df():
+    return pd.DataFrame(
+        {
+            "bigquery": ["string", "sql"],
+            "clickhouse": ["string", "sql"],
+            "dask": ["dataframe"],
+            "datafusion": ["sql"],
+            "druid": ["sqlalchemy", "sql"],
+            "duckdb": ["sqlalchemy", "sql"],
+            "impala": ["string", "sql"],
+            "mssql": ["sqlalchemy", "sql"],
+            "mysql": ["sqlalchemy", "sql"],
+            "oracle": ["sqlalchemy", "sql"],
+            "pandas": ["dataframe"],
+            "polars": ["dataframe"],
+            "postgres": ["sqlalchemy", "sql"],
+            "pyspark": ["dataframe"],
+            "snowflake": ["sqlalchemy", "sql"],
+            "sqlite": ["sqlalchemy", "sql"],
+            "trino": ["sqlalchemy", "sql"],
+        }.items(),
+        columns=["backend_name", "categories"],
+    )
+
+
+backend_info_table = ibis.memtable(backends_info_df())
+support_matrix_table = ibis.memtable(support_matrix_df())
+
+
+@st.cache_data(ttl=ONE_HOUR_IN_SECONDS)
+def get_all_backend_categories():
+    return (
+        backend_info_table.select(category=_.categories.unnest())
+        .distinct()
+        .order_by("category")["category"]
+        .to_pandas()
+        .tolist()
+    )
+
+
+@st.cache_data(ttl=ONE_HOUR_IN_SECONDS)
+def get_all_operation_categories():
+    return (
+        support_matrix_table.select(_.operation_category)
+        .distinct()["operation_category"]
+        .to_pandas()
+        .tolist()
+    )
+
+
+@st.cache_data(ttl=ONE_HOUR_IN_SECONDS)
+def get_backend_names(categories: Optional[list[str]] = None):
+    backend_expr = backend_info_table.mutate(category=_.categories.unnest())
+    if categories:
+        backend_expr = backend_expr.filter(_.category.isin(categories))
+    return (
+        backend_expr.select(_.backend_name).distinct().backend_name.to_pandas().tolist()
+    )
+
+
+def get_selected_backend_name():
+    backend_categories = get_all_backend_categories()
+    selected_categories_names = st.sidebar.multiselect(
+        "Backend category",
+        options=backend_categories,
+        default=backend_categories,
+    )
+    return get_backend_names(selected_categories_names)
+
+
+def get_backend_subset(subset):
+    return st.sidebar.multiselect("Backend name", options=subset, default=subset)
+
+
+def get_selected_operation_categories():
+    all_ops_categories = get_all_operation_categories()
+
+    selected_ops_categories = st.sidebar.multiselect(
+        "Operation category",
+        options=sorted(all_ops_categories),
+        default=None,
+    )
+    if not selected_ops_categories:
+        selected_ops_categories = all_ops_categories
+    show_geospatial = st.sidebar.checkbox("Include Geospatial ops", value=True)
+    if not show_geospatial and "geospatial" in selected_ops_categories:
+        selected_ops_categories.remove("geospatial")
+    return selected_ops_categories
+
+
+current_backend_names = get_backend_subset(get_selected_backend_name())
+sort_by_coverage = st.sidebar.checkbox("Sort by API Coverage", value=False)
+current_ops_categories = get_selected_operation_categories()
+
+hide_supported_by_all_backends = st.sidebar.selectbox(
+    "Operation compatibility",
+    ["Show all", "Show supported by all backends", "Hide supported by all backends"],
+    0,
+)
+show_full_ops_name = st.sidebar.checkbox("Show full operation name", False)
+
+# Start ibis expression
+table_expr = support_matrix_table
+
+# Add index to result
+if show_full_ops_name:
+    table_expr = table_expr.mutate(index=_.full_operation)
+else:
+    table_expr = table_expr.mutate(index=_.short_operation)
+table_expr = table_expr.order_by(_.index)
+
+# Filter operations by selected categories
+table_expr = table_expr.filter(_.operation_category.isin(current_ops_categories))
+
+# Filter operation by compatibility
+supported_backend_count = sum(
+    getattr(table_expr, backend_name).ifelse(1, 0)
+    for backend_name in current_backend_names
+)
+if hide_supported_by_all_backends == "Show supported by all backends":
+    table_expr = table_expr.filter(
+        supported_backend_count == len(current_backend_names)
+    )
+elif hide_supported_by_all_backends == "Hide supported by all backends":
+    table_expr = table_expr.filter(
+        supported_backend_count != len(current_backend_names)
+    )
+
+# Show only selected backend
+table_expr = table_expr[current_backend_names + ["index"]]
+
+# Execute query
+df = table_expr.to_pandas()
+df = df.set_index("index")
+
+# Display result
+all_visible_ops_count = len(df.index)
+if all_visible_ops_count:
+    # Compute coverage
+    coverage = (
+        df.sum()
+        .sort_values(ascending=False)
+        .map(lambda n: f"{n} ({round(100 * n / all_visible_ops_count)}%)")
+        .to_frame(name="API Coverage")
+        .T
+    )
+
+    table = pd.concat([coverage, df.replace({True: "✔", False: "🚫"})]).loc[
+        :, slice(None) if sort_by_coverage else sorted(df.columns)
+    ]
+    st.dataframe(table)
+else:
+    st.write("No data")
+
+with st.expander("SQL queries"):
+    for sql_query in sql_queries:
+        pretty_sql_query = sqlglot.transpile(
+            sql_query, read="duckdb", write="duckdb", pretty=True
+        )[0]
+        st.code(pretty_sql_query, language="sql")
+
+with st.expander("Source code"):
+    st.code(Path(__file__).read_text())
diff --git a/docs2/how-to/visualization/example_streamlit_app/example_streamlit_app.py b/docs2/how-to/visualization/example_streamlit_app/example_streamlit_app.py
new file mode 100644
--- /dev/null
+++ b/docs2/how-to/visualization/example_streamlit_app/example_streamlit_app.py
@@ -0,0 +1,72 @@
+from __future__ import annotations
+
+import requests
+import streamlit as st
+
+from ibis import _
+from ibis.streamlit import IbisConnection
+
+st.set_page_config(page_title="Yummy Data", layout="wide")
+st.title("Yummy Data :bacon:")
+
+
+@st.cache_data
+def get_emoji():
+    resp = requests.get(
+        "https://raw.githubusercontent.com/omnidan/node-emoji/master/lib/emoji.json"
+    )
+    resp.raise_for_status()
+    emojis = resp.json()
+    return emojis
+
+
+options = [1, 5, 10, 25, 50, 100]
+
+
+@st.cache_data
+def query():
+    return (
+        con.tables.recipes.relabel("snake_case")
+        .mutate(ner=_.ner.map(lambda n: n.lower()).unnest())
+        .ner.topk(max(options))
+        .relabel(dict(ner="ingredient"))
+        .to_pandas()
+        .assign(
+            emoji=lambda df: df.ingredient.map(
+                lambda emoji: f"{emojis.get(emoji, '-')}"
+            )
+        )
+        .set_index("ingredient")
+    )
+
+
+emojis = get_emoji()
+
+con = st.experimental_connection("ch", type=IbisConnection)
+
+if n := st.radio("Ingredients", options, index=1, horizontal=True):
+    table, whole = st.columns((2, 1))
+    idx = options.index(n)
+    k = 0
+    base = query()
+    for m in options[: idx + 1]:
+        df = base.iloc[k:m]
+        if not k:
+            word = "first"
+        elif m < n:
+            word = "next"
+        else:
+            word = "last"
+
+        uniq_emojis = " ".join(df.emoji[df.emoji != "-"].unique())
+        table.header(f"{word.title()} {m - k:d}")
+        table.subheader(uniq_emojis)
+
+        table.dataframe(df, use_container_width=True)
+        k = m
+
+    b = base.iloc[:n]
+    uniq_emojis = " ".join(b.emoji[b.emoji != "-"].unique())
+    whole.header(f"Top {n:d}")
+    whole.subheader(uniq_emojis)
+    whole.dataframe(b, use_container_width=True)
diff --git a/docs2/posts/old/rendered/model.py b/docs2/posts/old/rendered/model.py
new file mode 100644
--- /dev/null
+++ b/docs2/posts/old/rendered/model.py
@@ -0,0 +1,67 @@
+"""Linear regression model for predicting cab fares using PyTorch.
+
+Adapted from https://gist.github.com/pdet/e8d38734232c08e6c15aba79b4eb8368#file-taxi_prediction_example-py.
+"""
+from __future__ import annotations
+
+import pyarrow as pa
+import torch
+import tqdm
+from torch import nn
+
+
+class LinearRegression(nn.Module):
+    def __init__(self, input_dim, output_dim):
+        super().__init__()
+        self.linear = nn.Linear(input_dim, output_dim)
+
+    def forward(self, distances):
+        return self.linear(distances)
+
+
+class PredictCabFare:
+    def __init__(self, data, learning_rate: float = 0.01, epochs: int = 100) -> None:
+        # Define the input and output dimensions
+        input_dim = 1
+        output_dim = 1
+
+        # Create a linear regression model instance
+        self.data = data
+        self.model = LinearRegression(input_dim, output_dim)
+        self.learning_rate = learning_rate
+        self.epochs = epochs
+
+    def train(self):
+        distances = self.data["trip_distance"].reshape(-1, 1)
+        fares = self.data["fare_amount"].reshape(-1, 1)
+
+        # Define the loss function
+        criterion = nn.MSELoss()
+
+        # Define the optimizer
+        optimizer = torch.optim.SGD(self.model.parameters(), lr=self.learning_rate)
+
+        # Train the model
+        for _ in tqdm.trange(self.epochs):
+            # Forward pass
+            y_pred = self.model(distances)
+
+            # Compute loss
+            loss = criterion(y_pred, fares)
+
+            # Backward pass and optimize
+            optimizer.zero_grad()
+            loss.backward()
+            optimizer.step()
+
+    def predict(self, input):
+        with torch.no_grad():
+            return self.model(input)
+
+    def __call__(self, input: pa.ChunkedArray):
+        # Convert the input to numpy so it can be fed to the model
+        #
+        # .copy() to avoid the warning about undefined behavior from torch
+        input = torch.from_numpy(input.to_numpy().copy())[:, None]
+        predicted = self.predict(input).ravel()
+        return pa.array(predicted.numpy())


Test patch: