Repository: marimo-team/marimo
Pull request: update: update code block to ```python {.marimo}
fixes #1451

I just pushed this branch up from tinkering around back in September.

Outstanding:
  - [x] tests
    - [x] Basic backwards compat
  - [x] confirm `sql` blocks
  - [x] code highlight for "`python {.marimo}` on frontend

Problem statement:
Change md code-block style
### Description

Current codeblock style is  \```{.python.marimo}

I think code blocks should:
 - [x] syntax support in VScode out of the box (`{.python}` does the trick)
 - [ ] syntax support in github (missing)
 - [x] Have a loose check pattern (currently checks for just "{.*python.*}")
 - [x] Register as a quarto execution block
 - [x] Enable args (e.g. `{.python.marimo name="whatever"}`
 - [ ] Be pretty syntax (`{.python.marimo}` is a little weird)

### Suggested solution

> \```python {marimo}
 - [x] syntax support in VScode out of the box (`python` does the trick)
 - [x] syntax support in github (yep)
 - [x] Have a loose check pattern (maybe just check for "{marimo}")
 - [ ]  Register as a quarto execution block (this format breaks pandoc pretty bad)
 - [x] Enable args (e.g. `python {marimo name="whatever"}`
 - [x] Be pretty syntax (I like it!)

### Alternative

Making an issue for comments, but maybe better changed to a discussion post? I'm think asking on discord and getting feedback if people are actually using markdown.

### Additional context

@mscolnick suggestion!

Hints from issue discussion:
Others:


\```{marimo} python
```{marimo} python
assert None == 123, "string"
```

\```{python}
```{python}
assert None == 123, "string"
```

\```python .marimo (no good for pandoc)
```python
assert None == 123, "string"
```

---
Other suggestions:
 - Change github: https://github.com/github-linguist/linguist/blob/master/lib/linguist/languages.yml
 - Change pandoc: https://github.com/jgm/pandoc/blob/d01f2ac515ae3b744a476100f72050ca5446f328/src/Text/Pandoc/Readers/Markdown.hs#L684
Ok, just found this: https://github.com/jgm/pandoc/pull/8187

So these are fine

\```python {.marimo}
```python {.marimo}
assert None == 123, "string"
```

or 
\```python {#marimo}
```python {#marimo}
assert None == 123, "string"
```

or 
\```python {marimo=}
```python {marimo=}
assert None == 123, "string"
```

but still might be worth creating an issue for null attributes to make the \```python {marimo}
syntax work
I think i like `python {.marimo}` the best (unless can get `python {marimo}`)
The fix in pandoc looks pretty trivial: https://github.com/jgm/pandoc/issues/9791

I think it's going to be a matter of will- but I think it's good functionality. Hopefully, I get a positive response
One downside to this notation is that it breaks in marimo markdown itself (current renders no styling, but proposed does not render)

https://marimo.app/l/oaqlop

---

I might pick this up sooner than later with backwards compat in mind. As I'm working more directly with the notebooks (esp, with the kiosk mode picking up speed), the current notation is a little cumbersome

I don't think `python {marimo}` is happening anytime soon, so will put in `python {.marimo}`.
@mscolnick if the SQL changes are coming up, I can make sure `sql {.marimo}` works too
amazing! i agree it would be worth doing sooner to avoid too much breakage. If backwards compat is hard, we can make a breaking change and bump acccordingly (probably could read backwords compat and then write forwards)
Just checks to see if `{.*python.*}` is in the code block,

I think it should check for `{.*marimo.*}`. I think this will make it automatically backwards compatible?
Will submit PR once https://github.com/facelessuser/pymdown-extensions/pull/2470 is released, conditional on version number
not sure what the backwards compatibility issue is with the planned `python {.marimo}` code fence name. maybe using 

````
```python {.python.marimo}
print("python code")
```
```` 

could avoid the compatibility problem?

Commit message: update: update code block to ```python {.marimo}

Patch to review:
diff --git a/marimo/_cli/convert/markdown.py b/marimo/_cli/convert/markdown.py
index d04e843a4e9..05d04b90d52 100644
--- a/marimo/_cli/convert/markdown.py
+++ b/marimo/_cli/convert/markdown.py
@@ -22,22 +22,87 @@
     SuperFencesCodeExtension,
 )
 
+from marimo import _loggers
 from marimo._ast import codegen
 from marimo._ast.app import App, InternalApp, _AppConfig
 from marimo._ast.cell import Cell, CellConfig
 from marimo._ast.compiler import compile_cell
 from marimo._ast.names import DEFAULT_CELL_NAME
 from marimo._convert.utils import markdown_to_marimo
+from marimo._dependencies.dependencies import DependencyManager
+
+LOGGER = _loggers.marimo_logger()
 
 MARIMO_MD = "marimo-md"
 MARIMO_CODE = "marimo-code"
 
 ConvertKeys = Union[Literal["marimo"], Literal["marimo-app"]]
 
+# Regex captures loose yaml for frontmatter
+# Should match the following:
+# ---
+# title: "Title"
+# whatever
+# ---
+YAML_FRONT_MATTER_REGEX = re.compile(
+    r"^---\s*\n(.*?\n?)(?:---)\s*\n", re.UNICODE | re.DOTALL
+)
+
+# From pymdownx.superfences 10.11.2
+COMPAT_RE_NESTED_FENCE_START = re.compile(
+    r"""(?x)
+    (?P<fence>~{3,}|`{3,})
+    (?:[ \t]*\.?(?P<lang>[\w#.+-]+)(?=[\t ]|$))?                                           # Language
+    (?:
+        [ \t]*(\{(?P<attrs>[^\n]*)\}) |                                                    # Optional attributes or
+        (?P<options>
+            (?:
+                (?:[ \t]*[a-zA-Z][a-zA-Z0-9_]*(?:=(?P<quot>"|').*?(?P=quot))?)(?=[\t ]|$)  # Options
+            )+
+        )
+    )?[ \t]*$
+    """
+)
+
+
+def backwards_compatible_sanitization(line: str) -> str:
+    return line
+
+
+def extract_attribs(
+    line: str, fence_start: Optional[re.Match] = None
+) -> dict[str, str]:
+    # Extract attributes from the code block.
+    # Blocks are expected to be like this:
+    # python {.marimo disabled="true"}
+    if fence_start is None:
+        fence_start = RE_NESTED_FENCE_START.match(line)
+
+    if fence_start:
+        # attrs is a bit of a misnomer, matches
+        # .python.marimo disabled="true"
+        inner = fence_start.group("attrs")
+        if inner:
+            return dict(re.findall(r'(\w+)="([^"]*)"', inner))
+    return {}
+
 
 def _is_code_tag(text: str) -> bool:
     head = text.split("\n")[0].strip()
-    return bool(re.search(r"\{.*python.*\}", head))
+    legacy_format = bool(re.search(r"\{.*python.*\}", head))
+    legacy_format |= bool(re.search(r"\{.*sql.*\}", head))
+    if DependencyManager.new_superfences.has_required_version(quiet=True):
+        supported_format = bool(re.search(r".*\{.*marimo.*\}", head))
+        return legacy_format or supported_format
+    return legacy_format
+
+
+def _get_language(text: str) -> str:
+    header = text.split("\n").pop()
+    match = RE_NESTED_FENCE_START.match(header)
+    if match:
+        return match.group("lang")
+    return "python"
 
 
 def formatted_code_block(
@@ -46,14 +111,24 @@ def formatted_code_block(
     """Wraps code in a fenced code block with marimo attributes."""
     if attributes is None:
         attributes = {}
+    language = attributes.pop("language", "python")
     attribute_str = " ".join(
         [""] + [f'{key}="{value}"' for key, value in attributes.items()]
     )
     guard = "```"
     while guard in code:
         guard += "`"
+    if DependencyManager.new_superfences.has_required_version(quiet=True):
+        return "\n".join(
+            [
+                f"""{guard}{language} {{.marimo{attribute_str}}}""",
+                code,
+                guard,
+                "",
+            ]
+        )
     return "\n".join(
-        [f"""{guard}{{.python.marimo{attribute_str}}}""", code, guard, ""]
+        [f"""{guard}{{.{language}.marimo{attribute_str}}}""", code, guard, ""]
     )
 
 
@@ -82,6 +157,8 @@ def get_source_from_tag(tag: Element) -> str:
         if not (source and source.strip()):
             return ""
         source = markdown_to_marimo(source)
+    elif tag.attrib.get("language") == "sql":
+        return "#sql + " + source
     else:
         assert tag.tag == MARIMO_CODE, f"Unknown tag: {tag.tag}"
     return source
@@ -220,6 +297,10 @@ def __init__(self, *args: Any, **kwargs: Any) -> None:
         self.preprocessors.register(
             FrontMatterPreprocessor(self), "frontmatter", 100
         )
+        # Preprocess for backwards compatibility.
+        self.preprocessors.register(
+            MdCompatPreprocessor(self), "md-compat", 100
+        )
         fences_ext = SuperFencesCodeExtension()
         fences_ext.extendMarkdown(self)
         # TODO: Consider adding the admonition extension, and integrating it
@@ -272,15 +353,6 @@ def __init__(self, md: MarimoParser):
         super().__init__(md)
         self.md = md
         self.md.meta = {}
-        # Regex captures loose yaml for frontmatter
-        # Should match the following:
-        # ---
-        # title: "Title"
-        # whatever
-        # ---
-        self.yaml_front_matter_regex = re.compile(
-            r"^---\s*\n(.*?\n?)(?:---)\s*\n", re.UNICODE | re.DOTALL
-        )
 
     def run(self, lines: list[str]) -> list[str]:
         import yaml
@@ -295,7 +367,7 @@ def run(self, lines: list[str]) -> list[str]:
             return lines
 
         doc = "\n".join(lines)
-        result = self.yaml_front_matter_regex.match(doc)
+        result = YAML_FRONT_MATTER_REGEX.match(doc)
 
         if result:
             yaml_content = result.group(1)
@@ -310,6 +382,71 @@ def run(self, lines: list[str]) -> list[str]:
         return doc.split("\n")
 
 
+class MdCompatPreprocessor(Preprocessor):
+    """Preprocessor for backwards compatibility with old code blocks.
+
+    This preprocessor is used to convert old code blocks to the new format.
+    """
+
+    def run(self, lines: list[str]) -> list[str]:
+        response = []
+        old_to_old = False
+        old_to_new = False
+        for line in lines:
+            # TODO: Remove with some minor release.
+            if DependencyManager.new_superfences.has_required_version(
+                quiet=True
+            ):
+                response.append(line)
+                continue
+            # If old format, old regex, pass through but warn.
+            if RE_NESTED_FENCE_START.match(line):
+                old_to_old = True
+                response.append(line)
+                continue
+            # There's a chance that the new format is used with the old regex.
+            # since pymdownx.superfences < 10.11
+            # will not match
+            # ```lang {.marimo}
+            # put it into the old format, so super fences can handle it.
+            if new_match := COMPAT_RE_NESTED_FENCE_START.match(line):
+                old_to_new = True
+                attribute_str = " ".join(
+                    [""]
+                    + [
+                        f'{key}="{value}"'
+                        for key, value in extract_attribs(
+                            line, new_match
+                        ).items()
+                    ]
+                )
+                response.append(
+                    "".join(
+                        [
+                            new_match.group("fence"),
+                            "{",
+                            ".",
+                            new_match.group("lang"),
+                            ".marimo",
+                            attribute_str,
+                            "}",
+                        ]
+                    )
+                )
+                continue
+            response.append(line)
+        if old_to_old:
+            LOGGER.warning(
+                "Legacy format used for code block. Please update pymdownx to >= 10.11"
+            )
+
+        if old_to_new:
+            LOGGER.warning(
+                "Unsupported code fence, applying heuristic. Please update pymdownx to >= 10.11"
+            )
+        return response
+
+
 class SanitizeProcessor(Preprocessor):
     """Prevent unintended executable code block injection.
 
@@ -414,18 +551,12 @@ def add_paragraph() -> None:
             code_block = SubElement(parent, MARIMO_CODE)
             block_lines = code.split("\n")
             code_block.text = "\n".join(block_lines[1:-1])
-            # Extract attributes from the code block.
-            # Blocks are expected to be like this:
-            # {.python.marimo disabled="true"}
-            fence_start = RE_NESTED_FENCE_START.match(block_lines[0])
-            if fence_start:
-                # attrs is a bit of a misnomer, matches
-                # .python.marimo disabled="true"
-                inner = fence_start.group("attrs")
-                if inner:
-                    code_block.attrib = dict(
-                        re.findall(r'(\w+)="([^"]*)"', inner)
-                    )
+            code_block.set("language", _get_language(code))
+
+            attribs = extract_attribs(block_lines[0])
+            if attribs:
+                code_block.attrib = attribs
+
         add_paragraph()
         # Flush to indicate all blocks have been processed.
         blocks.clear()
diff --git a/marimo/_dependencies/dependencies.py b/marimo/_dependencies/dependencies.py
index 6afe5229e48..ecef28993bb 100644
--- a/marimo/_dependencies/dependencies.py
+++ b/marimo/_dependencies/dependencies.py
@@ -14,7 +14,7 @@ class Dependency:
     min_version: str | None = None
     max_version: str | None = None
 
-    def has(self) -> bool:
+    def has(self, quiet=False) -> bool:
         """Return True if the dependency is installed."""
         try:
             has_dep = importlib.util.find_spec(self.pkg) is not None
@@ -24,20 +24,32 @@ def has(self) -> bool:
             # Could happen for nested imports (e.g. foo.bar)
             return False
 
-        if self.min_version or self.max_version:
+        if not quiet and (self.min_version or self.max_version):
             self.warn_if_mismatch_version(self.min_version, self.max_version)
         return True
 
     def has_at_version(
-        self, *, min_version: str | None, max_version: str | None = None
+        self,
+        *,
+        min_version: str | None,
+        max_version: str | None = None,
+        quiet=False,
     ) -> bool:
-        if not self.has():
+        if not self.has(quiet=quiet):
             return False
         return _version_check(
             pkg=self.pkg,
             v=self.get_version(),
             min_v=min_version,
             max_v=max_version,
+            quiet=quiet,
+        )
+
+    def has_required_version(self, quiet=False) -> bool:
+        return self.has_at_version(
+            min_version=self.min_version,
+            max_version=self.max_version,
+            quiet=quiet,
         )
 
     def imported(self) -> bool:
@@ -75,7 +87,10 @@ def require_at_version(
         )
 
     def get_version(self) -> str:
-        return importlib.metadata.version(self.pkg)
+        try:
+            return importlib.metadata.version(self.pkg)
+        except importlib.metadata.PackageNotFoundError:
+            return __import__(self.pkg).__version__
 
     def warn_if_mismatch_version(
         self,
@@ -111,6 +126,7 @@ def _version_check(
     min_v: str | None = None,
     max_v: str | None = None,
     raise_error: bool = False,
+    quiet: bool = False,
 ) -> bool:
     if min_v is None and max_v is None:
         return True
@@ -125,14 +141,16 @@ def _version_check(
         msg = f"Mismatched version of {pkg}: expected >={min_v}, got {v}"
         if raise_error:
             raise RuntimeError(msg)
-        sys.stderr.write(f"{msg}. Some features may not work correctly.")
+        if not quiet:
+            sys.stderr.write(f"{msg}. Some features may not work correctly.")
         return False
 
     if parsed_max_version is not None and parsed_v >= parsed_max_version:
         msg = f"Mismatched version of {pkg}: expected <{max_v}, got {v}"
         if raise_error:
             raise RuntimeError(msg)
-        sys.stderr.write(f"{msg}. Some features may not work correctly.")
+        if not quiet:
+            sys.stderr.write(f"{msg}. Some features may not work correctly.")
         return False
 
     return True
@@ -170,6 +188,10 @@ class DependencyManager:
     groq = Dependency("groq")
     panel = Dependency("panel")
 
+    # Version requirements to properly support the new superfences introduced in
+    # pymdown#2470
+    new_superfences = Dependency("pymdownx", min_version="10.11.0")
+
     @staticmethod
     def has(pkg: str) -> bool:
         """Return True if any lib is installed."""
diff --git a/marimo/_output/md.py b/marimo/_output/md.py
index afb719d99b3..6789a28a95a 100644
--- a/marimo/_output/md.py
+++ b/marimo/_output/md.py
@@ -6,6 +6,9 @@
 
 import markdown  # type: ignore
 import pymdownx.emoji  # type: ignore
+from pymdownx.superfences import (  # type: ignore
+    SuperFencesCodeExtension,
+)
 
 from marimo._output.hypertext import Html
 from marimo._output.md_extensions.external_links import ExternalLinksExtension
@@ -108,6 +111,8 @@ def __init__(
                 ExternalLinksExtension(),
                 # Iconify
                 IconifyExtension(),
+                # Code fences for consistent syntax highlighting
+                SuperFencesCodeExtension(),
             ],
             extension_configs=extension_configs,  # type: ignore[arg-type]
         ).strip()
diff --git a/marimo/_server/export/exporter.py b/marimo/_server/export/exporter.py
index 592662de02a..09db2d4eab3 100644
--- a/marimo/_server/export/exporter.py
+++ b/marimo/_server/export/exporter.py
@@ -297,8 +297,6 @@ def _format_value(v: Optional[str | list[str]]) -> str | list[str]:
                 except SyntaxError:
                     pass
 
-            # Definitely no "cell"; as such, treat as code, as everything in
-            # marimo is code.
             if cell:
                 markdown = get_markdown_from_cell(cell, code)
                 # Unsanitized markdown is forced to code.
@@ -309,6 +307,8 @@ def _format_value(v: Optional[str | list[str]]) -> str | list[str]:
                     previous_was_markdown = True
                     document.append(markdown)
                     continue
+            # Definitely no "cell"; as such, treat as code, as everything in
+            # marimo is code.
             else:
                 attributes["unparsable"] = "true"
             # Add a blank line between markdown and code
diff --git a/marimo/_tutorials/markdown_format.md b/marimo/_tutorials/markdown_format.md
index cf0a688523d..b73a7ff1c8a 100644
--- a/marimo/_tutorials/markdown_format.md
+++ b/marimo/_tutorials/markdown_format.md
@@ -1,6 +1,6 @@
 ---
 title: Markdown
-marimo-version: 0.4.11
+marimo-version: 0.10.9
 ---
 
 # Markdown file format
@@ -35,14 +35,13 @@ by running the following command:
 $ marimo export md notebook.py > notebook.md
 ```
 <!---->
-
 ## Creating Python cells
 
 When you do need to create a Python cell in the markdown format, you can use a
 special code block:
 
 ````md
-```{.python.marimo}
+```python {.marimo}
 import matplotlib.pyplot as plt
 plt.plot([1, 2, 3, 4])
 ```
@@ -50,15 +49,15 @@ plt.plot([1, 2, 3, 4])
 
 This will create the following cell:
 
-```{.python.marimo}
+```python {.marimo}
 import matplotlib.pyplot as plt
+
 plt.plot([1, 2, 3, 4])
 plt.gca()
 ```
 
-As long as your code block contains the word `python` in a brace, like
-`{python}`, or `{.python note="Whatever you want"}`, marimo will treat it as a
-Python cell.
+As long as your code block contains the word `marimo` in a brace, like
+`{marimo}`, or `{.marimo note="Whatever you want"}`, marimo will treat it as a Python cell.
 ## `mo` tricks and tips
 
 You can break up markdown into multiple cells by using an empty html tag `<!---->`:
@@ -68,31 +67,32 @@ View the source of this notebook to see how this cell was created.
 You can still hide and disable code cells in markdown notebooks:
 
 ````md
-```{python hide_code="true"}
+```python {.marimo hide_code="true"}
 import pandas as pd
 pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})
 ```
 ````
 
-```{.python.marimo hide_code="true"}
+```python {.marimo hide_code="true"}
 import pandas as pd
-pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})
+
+pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
 ```
 
 ````md
-```{python disabled="true"}
+```python {.marimo disabled="true"}
 print("This code cell is disabled, there should be no output!")
 ```
 ````
 
-```{.python.marimo disabled="true"}
+```python {.marimo disabled="true"}
 print("This code cell is disabled, there should be no output!")
 ```
 
 Additionally, marimo knows when your code has a syntax issue:
 
 ````md
-```{python}
+```python {.marimo}
 print("This code cell has a syntax error"
 ```
 ````
@@ -100,12 +100,12 @@ print("This code cell has a syntax error"
 and on notebook save, will annotate the cell for you:
 
 ````md
-```{python unparseable="true"}
+```python {.marimo unparseable="true"}
 print("This code cell has a syntax error"
 ```
 ````
 
-```{.python.marimo unparsable="true"}
+```python {.marimo unparsable="true"}
 print("This code cell has a syntax error"
 ```
 
@@ -117,8 +117,8 @@ just give you the raw string. This lets you clearly delineate what values are
 supposed to be computed, and what values are static. To interpolate Python
 values, just use a Python cell:
 
-```{.python.marimo}
-'🍃' * 7
+```python {.marimo}
+"🍃" * 7
 ```
 
 ### Limitations on conversion
@@ -126,7 +126,7 @@ values, just use a Python cell:
 Whenever you try to implement a cell block like this:
 
 ````md
-```{python}
+```python {.marimo}
 mo.md("This is a markdown cell")
 ```
 ````
@@ -134,13 +134,15 @@ mo.md("This is a markdown cell")
 The markdown format will know to automatically keep this as markdown. However,
 some ambiguous cases can't be converted to markdown like this:
 
-````{.python.marimo}
-mo.md("""
-  This is a markdown cell with an execution block in it
-  ```{python}
-  # Too ambiguous to convert
-  ```
-  """)
+````python {.marimo}
+mo.md(
+    """
+    This is a markdown cell with an execution block in it
+    ```python {.marimo}
+    # Too ambiguous to convert
+    ```
+    """
+)
 ````
 
 It's not likely that you'll run into this issue, but rest assured that marimo
@@ -154,12 +156,12 @@ markdown notebook cells like you can in a python notebook; but you can still
 give your cells a name:
 
 ````md
-```{.python.marimo name="maybe"}
+```python {.marimo name="maybe"}
 # 🎵 Hey, I just met you, and this is crazy
 ```
 ````
 
-```{.python.marimo name="maybe"}
+```python {.marimo name="maybe"}
 # But here's my `cell_id`, so call me, `maybe` 🎶
 ```
 
@@ -177,6 +179,6 @@ $ marimo convert my_marimo.md > my_marimo.py
 Be sure to checkout the markdown.py tutorial (`marimo tutorial markdown`) for
 more information on to type-set and render markdown in marimo.
 
-```{.python.marimo hide_code="true"}
+```python {.marimo hide_code="true"}
 import marimo as mo
-```
+```
\ No newline at end of file


Code review comments:
diff_hunk: @@ -310,6 +382,71 @@ def run(self, lines: list[str]) -> list[str]:
         return doc.split("\n")
 
 
+class MdCompatPreprocessor(Preprocessor):
path: marimo/_cli/convert/markdown.py
text: @author:
I think this can just be removed, just have stricter version in pytoml

diff_hunk: @@ -108,6 +111,8 @@ def __init__(
                 ExternalLinksExtension(),
                 # Iconify
                 IconifyExtension(),
+                # Code fences for consistent syntax highlighting
path: marimo/_output/md.py
text: @author:
Might be able to parse without. double check