Repository: alecthomas/chroma
Pull request #1102 (2025-07-09 20:55:43)

Create lexer for Gemtext
Closes #560

Version 0.24.1 of the "Gemini hypertext format" is specified at <https://geminiprotocol.net/docs/gemtext-specification.gmi>.

This is my first time making a Lexer here :sweat_smile: I first built it in XML format, but couldn't work out how to get it to also handle syntax inside of preformatted blocks, so I moved it to Go using `markdown.go` as a reference.

<details>
<summary>my original <code>gemtext.xml</code>, which tests the same except preformatted blocks are always plain text</summary>

```xml
<lexer>
  <config>
    <name>Gemtext</name>
    <alias>gemtext</alias>
    <alias>gmi</alias>
    <alias>gmni</alias>
    <alias>gemini</alias>
    <filename>*.gmi</filename>
    <filename>*.gmni</filename>
    <filename>*.gemini</filename>
    <mime_type>text/gemini</mime_type>
  </config>
  <rules>
    <state name="root">
      <rule pattern="^(#[^#].+\r?\n)">
        <token type="GenericHeading" />
      </rule>
      <rule pattern="^(#{2,3}.+\r?\n)">
        <token type="GenericSubheading" />
      </rule>
      <rule pattern="^(\* )(.+\r?\n)">
        <bygroups>
          <token type="Keyword" />
          <token type="Text" />
        </bygroups>
      </rule>
      <rule pattern="^(>)(.+\r?\n)">
        <bygroups>
          <token type="Keyword" />
          <token type="GenericEmph" />
        </bygroups>
      </rule>
      <rule pattern="^(```\r?\n)([\w\W]*?)(^```)(.+\r?\n)?">
        <bygroups>
          <token type="LiteralString" />
          <token type="Text" />
          <token type="LiteralString" />
          <token type="Comment" />
        </bygroups>
      </rule>
      <rule pattern="^(```)(.+\r?\n)([\w\W]*?)(^```)(.+\r?\n)?">
        <bygroups>
          <token type="LiteralString" />
          <token type="LiteralString" />
          <token type="Text" />
          <token type="LiteralString" />
          <token type="Comment" />
        </bygroups>
      </rule>
      <rule pattern="^(=>)(\s*)([^\s]+)(\s*)$">
        <bygroups>
          <token type="Keyword" />
          <token type="Text" />
          <token type="NameAttribute" />
          <token type="Text" />
        </bygroups>
      </rule>
      <rule pattern="^(=>)(\s*)([^\s]+)(\s+)(.+)$">
        <bygroups>
          <token type="Keyword" />
          <token type="Text" />
          <token type="NameAttribute" />
          <token type="Text" />
          <token type="NameTag" />
        </bygroups>
      </rule>
      <rule pattern=".|(?:\r?\n)">
        <token type="Text" />
      </rule>
    </state>
  </rules>
</lexer>
```
</details>

The spec mentions that CRLF and LF are essentially interchangeable. I wasn't sure how to test that since IIRC Git clobbers CR on Unix systems, but this regex *seems* right at least..


Patch:
diff --git a/lexers/gemtext.go b/lexers/gemtext.go
new file mode 100644
index 0000000..2fed8d6
--- /dev/null
+++ b/lexers/gemtext.go
@@ -0,0 +1,37 @@
+package lexers
+
+import (
+	. "github.com/alecthomas/chroma/v2" // nolint
+)
+
+// Gemtext lexer.
+var Gemtext = Register(MustNewLexer(
+	&Config{
+		Name:      "Gemtext",
+		Aliases:   []string{"gemtext", "gmi", "gmni", "gemini"},
+		Filenames: []string{"*.gmi", "*.gmni", "*.gemini"},
+		MimeTypes: []string{"text/gemini"},
+	},
+	gemtextRules,
+))
+
+func gemtextRules() Rules {
+	return Rules{
+		"root": {
+			{`^(#[^#].+\r?\n)`, ByGroups(GenericHeading), nil},
+			{`^(#{2,3}.+\r?\n)`, ByGroups(GenericSubheading), nil},
+			{`^(\* )(.+\r?\n)`, ByGroups(Keyword, Text), nil},
+			{`^(>)(.+\r?\n)`, ByGroups(Keyword, GenericEmph), nil},
+			{"^(```\\r?\\n)([\\w\\W]*?)(^```)(.+\\r?\\n)?", ByGroups(String, Text, String, Comment), nil},
+			{
+				"^(```)(\\w+)(\\r?\\n)([\\w\\W]*?)(^```)(.+\\r?\\n)?",
+				UsingByGroup(2, 4, String, String, String, Text, String, Comment),
+				nil,
+			},
+			{"^(```)(.+\\r?\\n)([\\w\\W]*?)(^```)(.+\\r?\\n)?", ByGroups(String, String, Text, String, Comment), nil},
+			{`^(=>)(\s*)([^\s]+)(\s*)$`, ByGroups(Keyword, Text, NameAttribute, Text), nil},
+			{`^(=>)(\s*)([^\s]+)(\s+)(.+)$`, ByGroups(Keyword, Text, NameAttribute, Text, NameTag), nil},
+			{`(.|(?:\r?\n))`, ByGroups(Text), nil},
+		},
+	}
+}


Test patch:
diff --git a/lexers/testdata/gemtext.actual b/lexers/testdata/gemtext.actual
new file mode 100644
index 0000000..a3d2690
--- /dev/null
+++ b/lexers/testdata/gemtext.actual
@@ -0,0 +1,90 @@
+# This is a heading!
+
+#This is also a heading!
+
+This is some text. It is not special.
+
+## Another heading
+
+##yes, leading spaces are optional
+
+=> gemini://geminiprotocol.net/docs/gemtext-specification.gmi
+
+=> gemini://geminiprotocol.net/docs/gemtext-specification.gmi Here is a friendly link to the Gemtext specification
+
+=> https://geminiprotocol.net/docs/gemtext-specification.gmi gemini protocol isn't special for links
+
+=> gemini://geminiprotocol.net/	multiple kinds of whitespace
+
+=>  gemini://geminiprotocol.net/	 multiple whitespace between parts!
+
+=> /foo/bar/baz.txt leading slash still counts
+
+Links may also be closer together. Here are some examples from the spec:
+=> gemini://example.org/
+=> gemini://example.org/ An example link
+=> gemini://example.org/foo	Another example link at the same host
+=> foo/bar/baz.txt	A relative link
+=> 	gopher://example.org:70/1 A gopher link
+
+=>this shouldn't be a link, but it is. 
+=>gemini://example.org/ the leading space is optional
+
+### Why not try lists?
+
+
+###spaces are still optional
+
+* This is a list item.
+* Here's another.
+*  Still a list item!
+*This is not.
+** This is also not.
+*	Neither is this.
+- Not a list item.
+  - Not a list item.
+
+> Someone said this.
+
+>Someone also said this.
+
+```
+This text is preformatted.
+
+# Hello, world!
+
+This isn't to be treated as Gemtext:
+=> gemini://example.com
+
+> no one said this
+```
+
+```
+This is also plaintext
+```This text is ignored
+
+
+```This is alt text
+This text is also preformatted.
+```this text is ignored.
+
+```Art by Joan Stark of a camp site. A small tent faces a small campfire. There is a log nearby, perfect for sitting on. The initials "jgs" can be seen.
+        ______
+jgs    /     /\
+      /     /  \
+     /_____/----\_    (
+    "     "          ).
+   _ ___          o (:') o
+  (@))_))        o ~/~~\~ o
+                  o  o  o
+```
+
+Syntax highlighting may be applied to preformatted blocks:
+```javascript
+	column.substring(0,num)
+```
+
+The spec says, "Any text following the leading "```" of a preformat toggle line MUST be ignored by clients." So...
+```
+```this is a comment!
+wow, what WAS that??
diff --git a/lexers/testdata/gemtext.expected b/lexers/testdata/gemtext.expected
new file mode 100644
index 0000000..0b35d1f
--- /dev/null
+++ b/lexers/testdata/gemtext.expected
@@ -0,0 +1,132 @@
+[
+  {"type":"GenericHeading","value":"# This is a heading!\n"},
+  {"type":"Text","value":"\n"},
+  {"type":"GenericHeading","value":"#This is also a heading!\n"},
+  {"type":"Text","value":"\nThis is some text. It is not special.\n\n"},
+  {"type":"GenericSubheading","value":"## Another heading\n"},
+  {"type":"Text","value":"\n"},
+  {"type":"GenericSubheading","value":"##yes, leading spaces are optional\n"},
+  {"type":"Text","value":"\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"Text","value":" "},
+  {"type":"NameAttribute","value":"gemini://geminiprotocol.net/docs/gemtext-specification.gmi"},
+  {"type":"Text","value":"\n\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"Text","value":" "},
+  {"type":"NameAttribute","value":"gemini://geminiprotocol.net/docs/gemtext-specification.gmi"},
+  {"type":"Text","value":" "},
+  {"type":"NameTag","value":"Here is a friendly link to the Gemtext specification"},
+  {"type":"Text","value":"\n\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"Text","value":" "},
+  {"type":"NameAttribute","value":"https://geminiprotocol.net/docs/gemtext-specification.gmi"},
+  {"type":"Text","value":" "},
+  {"type":"NameTag","value":"gemini protocol isn't special for links"},
+  {"type":"Text","value":"\n\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"Text","value":" "},
+  {"type":"NameAttribute","value":"gemini://geminiprotocol.net/"},
+  {"type":"Text","value":"\t"},
+  {"type":"NameTag","value":"multiple kinds of whitespace"},
+  {"type":"Text","value":"\n\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"Text","value":"  "},
+  {"type":"NameAttribute","value":"gemini://geminiprotocol.net/"},
+  {"type":"Text","value":"\t "},
+  {"type":"NameTag","value":"multiple whitespace between parts!"},
+  {"type":"Text","value":"\n\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"Text","value":" "},
+  {"type":"NameAttribute","value":"/foo/bar/baz.txt"},
+  {"type":"Text","value":" "},
+  {"type":"NameTag","value":"leading slash still counts"},
+  {"type":"Text","value":"\n\nLinks may also be closer together. Here are some examples from the spec:\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"Text","value":" "},
+  {"type":"NameAttribute","value":"gemini://example.org/"},
+  {"type":"Text","value":"\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"Text","value":" "},
+  {"type":"NameAttribute","value":"gemini://example.org/"},
+  {"type":"Text","value":" "},
+  {"type":"NameTag","value":"An example link"},
+  {"type":"Text","value":"\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"Text","value":" "},
+  {"type":"NameAttribute","value":"gemini://example.org/foo"},
+  {"type":"Text","value":"\t"},
+  {"type":"NameTag","value":"Another example link at the same host"},
+  {"type":"Text","value":"\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"Text","value":" "},
+  {"type":"NameAttribute","value":"foo/bar/baz.txt"},
+  {"type":"Text","value":"\t"},
+  {"type":"NameTag","value":"A relative link"},
+  {"type":"Text","value":"\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"Text","value":" \t"},
+  {"type":"NameAttribute","value":"gopher://example.org:70/1"},
+  {"type":"Text","value":" "},
+  {"type":"NameTag","value":"A gopher link"},
+  {"type":"Text","value":"\n\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"NameAttribute","value":"this"},
+  {"type":"Text","value":" "},
+  {"type":"NameTag","value":"shouldn't be a link, but it is. "},
+  {"type":"Text","value":"\n"},
+  {"type":"Keyword","value":"=\u003e"},
+  {"type":"NameAttribute","value":"gemini://example.org/"},
+  {"type":"Text","value":" "},
+  {"type":"NameTag","value":"the leading space is optional"},
+  {"type":"Text","value":"\n\n"},
+  {"type":"GenericSubheading","value":"### Why not try lists?\n"},
+  {"type":"Text","value":"\n\n"},
+  {"type":"GenericSubheading","value":"###spaces are still optional\n"},
+  {"type":"Text","value":"\n"},
+  {"type":"Keyword","value":"* "},
+  {"type":"Text","value":"This is a list item.\n"},
+  {"type":"Keyword","value":"* "},
+  {"type":"Text","value":"Here's another.\n"},
+  {"type":"Keyword","value":"* "},
+  {"type":"Text","value":" Still a list item!\n*This is not.\n** This is also not.\n*\tNeither is this.\n- Not a list item.\n  - Not a list item.\n\n"},
+  {"type":"Keyword","value":"\u003e"},
+  {"type":"GenericEmph","value":" Someone said this.\n"},
+  {"type":"Text","value":"\n"},
+  {"type":"Keyword","value":"\u003e"},
+  {"type":"GenericEmph","value":"Someone also said this.\n"},
+  {"type":"Text","value":"\n"},
+  {"type":"LiteralString","value":"```\n"},
+  {"type":"Text","value":"This text is preformatted.\n\n# Hello, world!\n\nThis isn't to be treated as Gemtext:\n=\u003e gemini://example.com\n\n\u003e no one said this\n"},
+  {"type":"LiteralString","value":"```"},
+  {"type":"Text","value":"\n\n"},
+  {"type":"LiteralString","value":"```\n"},
+  {"type":"Text","value":"This is also plaintext\n"},
+  {"type":"LiteralString","value":"```"},
+  {"type":"Comment","value":"This text is ignored\n"},
+  {"type":"Text","value":"\n\n"},
+  {"type":"LiteralString","value":"```This is alt text\n"},
+  {"type":"Text","value":"This text is also preformatted.\n"},
+  {"type":"LiteralString","value":"```"},
+  {"type":"Comment","value":"this text is ignored.\n"},
+  {"type":"Text","value":"\n"},
+  {"type":"LiteralString","value":"```Art by Joan Stark of a camp site. A small tent faces a small campfire. There is a log nearby, perfect for sitting on. The initials \"jgs\" can be seen.\n"},
+  {"type":"Text","value":"        ______\njgs    /     /\\\n      /     /  \\\n     /_____/----\\_    (\n    \"     \"          ).\n   _ ___          o (:') o\n  (@))_))        o ~/~~\\~ o\n                  o  o  o\n"},
+  {"type":"LiteralString","value":"```"},
+  {"type":"Text","value":"\n\nSyntax highlighting may be applied to preformatted blocks:\n"},
+  {"type":"LiteralString","value":"```javascript\n"},
+  {"type":"Text","value":"\t"},
+  {"type":"NameOther","value":"column"},
+  {"type":"Punctuation","value":"."},
+  {"type":"NameOther","value":"substring"},
+  {"type":"Punctuation","value":"("},
+  {"type":"LiteralNumberInteger","value":"0"},
+  {"type":"Punctuation","value":","},
+  {"type":"NameOther","value":"num"},
+  {"type":"Punctuation","value":")"},
+  {"type":"Text","value":"\n"},
+  {"type":"LiteralString","value":"```"},
+  {"type":"Text","value":"\n\nThe spec says, \"Any text following the leading \"```\" of a preformat toggle line MUST be ignored by clients.\" So...\n"},
+  {"type":"LiteralString","value":"```\n```"},
+  {"type":"Comment","value":"this is a comment!\n"},
+  {"type":"Text","value":"wow, what WAS that??\n"}
+]