Repository: etcd-io/etcd (language: go)

Issue:
Find a replacement for marker (markdown linter)
### What would you like to be added?

Right now we're using [marker](https://github.com/crawford/marker) as the markdown linter. However, I see two issues:

1. Its functionality is limited, from its `README`
    > This is a tool for finding issues in [CommonMark](http://commonmark.org/) documentation. Right now, it only identifies broken links (malformed URLs, non-existent paths, etc.).
2. It's written in Rust, and while dropping our GitHub action for static checking (#18058). We wont run this linter in the Prow infrastructure as we don't have Rust/Cargo available (refer to https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/etcd-io_etcd/18058/pull-etcd-verify/1793352130300481536#1:build-log.txt%3A659-662).

While trying to fix 2, I opened a PR crawford/marker#32 to be able to use a published binary, rather than building the project itself.

However, it would be better to use a Go-written Markdown linter as alternative.

### Why is this needed?

To keep linting markdown files.


Discussion / hints:
cc. @jmhbnz 
@ivanvc  Can we use markdownlint which is officially on Go.pkg.dev : https://pkg.go.dev/github.com/svendowideit/markdownlint
https://github.com/SvenDowideit/markdownlint/blob/v0.9.4/main.go
> @ivanvc Can we use markdownlint which is officially on Go.pkg.dev : https://pkg.go.dev/github.com/svendowideit/markdownlint
> https://github.com/SvenDowideit/markdownlint/blob/v0.9.4/main.go

It looks like an abandoned project. It was originally done by Docker, but it's now a public archive, and its last commit is from 2017. I think, ideally we should replace it with other linter that has more wide adoption.

Their README mentions that it was in use for https://docs.docker.com. However, Docker docs is now using NodeJS' markdownlint: 
* https://github.com/docker/docs/blob/b952723881c6e6cd7dcc556ab4c1d5a1818ec08d/package.json#L33
* https://github.com/docker/docs/blob/b952723881c6e6cd7dcc556ab4c1d5a1818ec08d/Dockerfile#L35
Surveying GitHub doesn't seem to be promising. There are some projects, but they seem unmaintained, or abandoned: https://github.com/search?q=markdown+linter+language%3AGo&type=repositories
> It's written in Rust

IMO, it's not related to RUST. It needs a release to publish executable binary. For Go ecosystem, `go install` can help us. However, it doesn't mean we should replace it. Just my two cents.

As far as I know, the `marker` is good one for the linter. I was thinking about how to add makefile to install it without `cargo`, mentioned by https://github.com/etcd-io/etcd/pull/16708#issuecomment-1761226782. 

Is it possible to be considered as an option? upload binary to storage which support anonymous download, like 

https://github.com/etcd-io/etcd/blob/bb701b9265f31d61db5906325e0a7e2abf7d3627/scripts/install-marker.sh#L12-L15

Thanks,
Wei
Hey @fuweid, I agree. After researching a native-Go alternative and yielding no results, I'm leaning towards having a linter that we can use as a binary. I opened PR https://github.com/crawford/marker/pull/32 to have `marker` distributed as a binary. Therefore, we won't need to have cargo installed locally, nor do we need to host the compiled binary (https://github.com/etcd-io/etcd/pull/16708#issuecomment-1761226782).

However, my original point is that the use-case for `marker` is limited; as it states on its `README`, it only checks broken links. It doesn't do any other Markdown linting:

> This is a tool for finding issues in CommonMark documentation. Right now, it only identifies broken links (malformed URLs, non-existent paths, etc.).
@ivanvc Nice!

> However, my original point is that the use-case for marker is limited; as it states on its README, it only checks broken links. It doesn't do any other Markdown linting:

If we do have new linter requirement, maybe we can file pull request to `marker`. The detector for broken linters is common use case because it's easy to break it by refactor of docs. 
With #18318 merged, I was looking into cleaning up `scripts/install-marker.sh`. But it looks like it's only used by `tests/Dockerfile`. Is this a `Dockerfile` we want to maintain? The Ubuntu version used is no longer maintained (21.10). Building the image fails because the APT repositories don't work anymore.

@jmhbnz, thoughts? (or @ahrtr, as I see you were the last to commit to that `Dockerfile`).
> But it looks like it's only used by `tests/Dockerfile`. Is this a `Dockerfile` we want to maintain?

I am not aware of any usage on the `tests/Dockerfile`. Probably we can just remove it.
Are we okay with keeping marker as the markdown linter, considering it only checks for broken links? Do we want or need to do more linting, considering that running our Markdown files against NodeJS' `markdownlint-cli` with the default configuration raises too many warnings?
Discussed during sig-etcd triage meeting. We recently put work into updating `etcd-io/website` to put a proper markdown linting workflow in place.

I am supportive of adding a new prow presubmit for `etcd-io/etcd` to do the same.

Once the new workflow is in place I would suggest we use a good first issue to get help from the community to address the backlog of markdown files that need updates.

Edit: For a detailed next step, we need to replicate the [script](https://github.com/etcd-io/website/blob/main/scripts/markdown_diff_lint.sh) and [Makefile recipe](https://github.com/etcd-io/website/blob/main/Makefile#L31-L33) for markdown linting from `etcd-io/website` to `etcd-io/etcd` Once that is done we could create the new prow job.
Hi, can I look into this?
Hi, I would also love to try this out. 
The first step is to set the foundation to start the markdown linting. As a reference, we're already doing it on the website (as noted in https://github.com/etcd-io/etcd/issues/18059#issuecomment-2436062946).

After we have the `Makefile` target, we'll need to do a new prow job, taking the website's as reference: https://github.com/kubernetes/test-infra/blob/7c58ef243f8845838f1b545c140a8ed615516441/config/jobs/etcd/etcd-website-presubmits.yaml

@Zanda256, @joshjms, we usually run FIFO. Given that you both want to contribute, we can split the work on doing the `Makefile` assigned to @Zanda256. And after the `Makefile` target is in place, @joshjms can work on the Prow job. Does that sound good?
Sounds good. Is that okay with you @Zanda256 ?
Sure @joshjms , I'm okay with that @ivanvc .
🎉 

/assign @Zanda256 @joshjms 

Fix patch:
diff --git a/Makefile b/Makefile
index 25da105a38e..20df43d29e5 100644
--- a/Makefile
+++ b/Makefile
@@ -218,3 +218,8 @@ verify-go-versions:
 .PHONY: sync-toolchain-directive
 sync-toolchain-directive:
 	./scripts/sync_go_toolchain_directive.sh
+
+.PHONY: markdown-diff-lint
+markdown-diff-lint:
+	./scripts/markdown_diff_lint.sh
+	
\ No newline at end of file
diff --git a/scripts/markdown_diff_lint.sh b/scripts/markdown_diff_lint.sh
new file mode 100755
index 00000000000..d864d3e7e1f
--- /dev/null
+++ b/scripts/markdown_diff_lint.sh
@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+# This script runs markdownlint-cli2 on changed files.
+# Usage: ./markdown_lint.sh
+
+
+# We source ./scripts/test_lib.sh, it sets the log functions and color variables.
+source ./scripts/test_lib.sh
+
+# When we source ./scripts/test_lib.sh, it has the line set -u which treats unset variables as errors.
+# We need to unset the variable to avoid the error.
+set +u -eo pipefail
+
+if ! command markdownlint-cli2 dummy.md &>/dev/null; then
+  log_error "markdownlint-cli2 needs to be installed."
+  log_error "Please refer to https://github.com/DavidAnson/markdownlint-cli2?tab=readme-ov-file#install for installation instructions."
+  exit 1
+fi
+
+
+if [ -z "${PULL_BASE_SHA}" ]; then
+  echo "Empty base reference (\$PULL_BASE_SHA), assuming: main"
+  PULL_BASE_SHA=main
+fi
+
+if [ -z "${PULL_PULL_SHA}" ]; then
+  PULL_PULL_SHA="$(git rev-parse HEAD)"
+  echo "Empty pull reference (\$PULL_PULL_SHA), assuming: ${PULL_PULL_SHA}"
+fi
+
+MD_LINT_URL_PREFIX="https://github.com/DavidAnson/markdownlint/blob/main/doc/"
+
+mapfile -t changed_files < <(git diff "${PULL_BASE_SHA}" --name-only)
+declare -A files_with_failures start_ranges end_ranges
+
+for file in "${changed_files[@]}"; do
+  if ! [[ "$file" =~ .md$ ]]; then
+    continue
+  fi
+
+  # Find start and end ranges from changed files.
+  start_ranges=()
+  end_ranges=()
+  # From https://github.com/paleite/eslint-plugin-diff/blob/46c5bcf296e9928db19333288457bf2805aad3b9/src/git.ts#L8-L27
+  ranges=$(git diff "${PULL_BASE_SHA}" \
+           --diff-algorithm=histogram \
+           --diff-filter=ACM \
+           --find-renames=100% \
+           --no-ext-diff \
+           --relative \
+           --unified=0 -- "${file}" | \
+    gawk 'match($0, /^@@\s-[0-9,]+\s\+([0-9]+)(,([0-9]+))?/, m) { \
+               print m[1] ":" m[1] + ((m[3] == "") ? "0" : m[3]) }')
+  i=0
+  for range in ${ranges}; do
+    start_ranges["${i}"]=$(echo "${range}" | awk -F: '{print $1}')
+    end_ranges["${i}"]=$(echo "${range}" | awk -F: '{print $2}')
+    i=$((1 + i))
+  done
+  if [ -z "${ranges}" ]; then
+    start_ranges[0]=0
+    end_ranges[0]=0
+  fi
+
+  i=0
+  # Run markdownlint-cli2 with the changed file and print only the summary (stdout).
+  markdownlint-cli2 "${file}" --config "${ETCD_ROOT_DIR}/tools/.markdownlint.jsonc" 2>/dev/null || true
+  while IFS= read -r line; do
+    line_number=$(echo "${line}" | awk -F: '{print $2}' | awk '{print $1}')
+    while [ "${i}" -lt "${#end_ranges[@]}" ] && [ "${line_number}" -gt "${end_ranges["${i}"]}" ]; do
+      i=$((1 + i))
+    done
+    rule=$(echo "${line}" | gawk 'match($2, /([^\/]+)/, m) {print tolower(m[1])}')
+    lint_error="${line} (${MD_LINT_URL_PREFIX}${rule}.md)"
+
+    if [ "${i}" -lt "${#start_ranges[@]}" ] && [ "${line_number}" -ge "${start_ranges["${i}"]}" ] && [ "${line_number}" -le "${end_ranges["${i}"]}" ]; then
+      # Inside range with changes, raise an error.
+      log_error "${lint_error}"
+      files_with_failures["${file}"]=1
+    else
+      # Outside of range, raise a warning.
+      log_warning "${lint_error}"
+    fi
+  done < <(markdownlint-cli2 "${file}" --config "${ETCD_ROOT_DIR}/tools/.markdownlint.jsonc" 2>&1 >/dev/null || true)
+done
+
+echo "Finished linting"
+
+for file in "${!files_with_failures[@]}"; do
+  log_error "${file} has linting issues"
+done
+if [ "${#files_with_failures[@]}" -gt "0" ]; then
+  exit 1
+fi
+
diff --git a/tools/.markdownlint.jsonc b/tools/.markdownlint.jsonc
new file mode 100644
index 00000000000..0263558de6d
--- /dev/null
+++ b/tools/.markdownlint.jsonc
@@ -0,0 +1,310 @@
+// Example markdownlint configuration with all properties set to their default value
+{
+
+    // Default state for all rules
+    "default": true,
+  
+    // Path to configuration file to extend
+    "extends": null,
+  
+    // MD001/heading-increment : Heading levels should only increment by one level at a time : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md001.md
+    "MD001": true,
+  
+    // MD003/heading-style : Heading style : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md003.md
+    "MD003": {
+      // Heading style
+      "style": "consistent"
+    },
+  
+    // MD004/ul-style : Unordered list style : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md004.md
+    "MD004": {
+      // List style
+      "style": "consistent"
+    },
+  
+    // MD005/list-indent : Inconsistent indentation for list items at the same level : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md005.md
+    "MD005": true,
+  
+    // MD007/ul-indent : Unordered list indentation : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md007.md
+    "MD007": {
+      // Spaces for indent
+      "indent": 2,
+      // Whether to indent the first level of the list
+      "start_indented": false,
+      // Spaces for first level indent (when start_indented is set)
+      "start_indent": 2
+    },
+  
+    // MD009/no-trailing-spaces : Trailing spaces : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md009.md
+    "MD009": {
+      // Spaces for line break
+      "br_spaces": 2,
+      // Allow spaces for empty lines in list items
+      "list_item_empty_lines": false,
+      // Include unnecessary breaks
+      "strict": false
+    },
+  
+    // MD010/no-hard-tabs : Hard tabs : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md010.md
+    "MD010": {
+      // Include code blocks
+      "code_blocks": true,
+      // Fenced code languages to ignore
+      "ignore_code_languages": [],
+      // Number of spaces for each hard tab
+      "spaces_per_tab": 1
+    },
+  
+    // MD011/no-reversed-links : Reversed link syntax : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md011.md
+    "MD011": true,
+  
+    // MD012/no-multiple-blanks : Multiple consecutive blank lines : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md012.md
+    "MD012": {
+      // Consecutive blank lines
+      "maximum": 1
+    },
+  
+    // MD013/line-length : Line length : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md013.md
+    "MD013": {
+      // Number of characters
+      "line_length": 80,
+      // Number of characters for headings
+      "heading_line_length": 80,
+      // Number of characters for code blocks
+      "code_block_line_length": 80,
+      // Include code blocks
+      "code_blocks": true,
+      // Include tables
+      "tables": true,
+      // Include headings
+      "headings": true,
+      // Strict length checking
+      "strict": false,
+      // Stern length checking
+      "stern": false
+    },
+  
+    // MD014/commands-show-output : Dollar signs used before commands without showing output : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md014.md
+    "MD014": true,
+  
+    // MD018/no-missing-space-atx : No space after hash on atx style heading : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md018.md
+    "MD018": true,
+  
+    // MD019/no-multiple-space-atx : Multiple spaces after hash on atx style heading : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md019.md
+    "MD019": true,
+  
+    // MD020/no-missing-space-closed-atx : No space inside hashes on closed atx style heading : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md020.md
+    "MD020": true,
+  
+    // MD021/no-multiple-space-closed-atx : Multiple spaces inside hashes on closed atx style heading : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md021.md
+    "MD021": true,
+  
+    // MD022/blanks-around-headings : Headings should be surrounded by blank lines : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md022.md
+    "MD022": {
+      // Blank lines above heading
+      "lines_above": 1,
+      // Blank lines below heading
+      "lines_below": 1
+    },
+  
+    // MD023/heading-start-left : Headings must start at the beginning of the line : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md023.md
+    "MD023": true,
+  
+    // MD024/no-duplicate-heading : Multiple headings with the same content : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md024.md
+    "MD024": {
+      // Only check sibling headings
+      "siblings_only": false
+    },
+  
+    // MD025/single-title/single-h1 : Multiple top-level headings in the same document : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md025.md
+    "MD025": {
+      // Heading level
+      "level": 1,
+      // RegExp for matching title in front matter
+      "front_matter_title": "^\\s*title\\s*[:=]"
+    },
+  
+    // MD026/no-trailing-punctuation : Trailing punctuation in heading : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md026.md
+    "MD026": {
+      // Punctuation characters
+      "punctuation": ".,;:!。，；：！"
+    },
+  
+    // MD027/no-multiple-space-blockquote : Multiple spaces after blockquote symbol : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md027.md
+    "MD027": true,
+  
+    // MD028/no-blanks-blockquote : Blank line inside blockquote : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md028.md
+    "MD028": true,
+  
+    // MD029/ol-prefix : Ordered list item prefix : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md029.md
+    "MD029": {
+      // List style
+      "style": "one_or_ordered"
+    },
+  
+    // MD030/list-marker-space : Spaces after list markers : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md030.md
+    "MD030": {
+      // Spaces for single-line unordered list items
+      "ul_single": 1,
+      // Spaces for single-line ordered list items
+      "ol_single": 1,
+      // Spaces for multi-line unordered list items
+      "ul_multi": 1,
+      // Spaces for multi-line ordered list items
+      "ol_multi": 1
+    },
+  
+    // MD031/blanks-around-fences : Fenced code blocks should be surrounded by blank lines : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md031.md
+    "MD031": {
+      // Include list items
+      "list_items": true
+    },
+  
+    // MD032/blanks-around-lists : Lists should be surrounded by blank lines : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md032.md
+    "MD032": true,
+  
+    // MD033/no-inline-html : Inline HTML : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md033.md
+    "MD033": {
+      // Allowed elements
+      "allowed_elements": []
+    },
+  
+    // MD034/no-bare-urls : Bare URL used : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md034.md
+    "MD034": true,
+  
+    // MD035/hr-style : Horizontal rule style : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md035.md
+    "MD035": {
+      // Horizontal rule style
+      "style": "consistent"
+    },
+  
+    // MD036/no-emphasis-as-heading : Emphasis used instead of a heading : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md036.md
+    "MD036": {
+      // Punctuation characters
+      "punctuation": ".,;:!?。，；：！？"
+    },
+  
+    // MD037/no-space-in-emphasis : Spaces inside emphasis markers : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md037.md
+    "MD037": true,
+  
+    // MD038/no-space-in-code : Spaces inside code span elements : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md038.md
+    "MD038": true,
+  
+    // MD039/no-space-in-links : Spaces inside link text : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md039.md
+    "MD039": true,
+  
+    // MD040/fenced-code-language : Fenced code blocks should have a language specified : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md040.md
+    "MD040": {
+      // List of languages
+      "allowed_languages": [],
+      // Require language only
+      "language_only": false
+    },
+  
+    // MD041/first-line-heading/first-line-h1 : First line in a file should be a top-level heading : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md041.md
+    "MD041": {
+      // Heading level
+      "level": 1,
+      // RegExp for matching title in front matter
+      "front_matter_title": "^\\s*title\\s*[:=]"
+    },
+  
+    // MD042/no-empty-links : No empty links : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md042.md
+    "MD042": true,
+  
+    // MD043/required-headings : Required heading structure : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md043.md
+    "MD043": {
+      // List of headings
+      "headings": [],
+      // Match case of headings
+      "match_case": false
+    },
+  
+    // MD044/proper-names : Proper names should have the correct capitalization : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md044.md
+    "MD044": {
+      // List of proper names
+      "names": [],
+      // Include code blocks
+      "code_blocks": true,
+      // Include HTML elements
+      "html_elements": true
+    },
+  
+    // MD045/no-alt-text : Images should have alternate text (alt text) : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md045.md
+    "MD045": true,
+  
+    // MD046/code-block-style : Code block style : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md046.md
+    "MD046": {
+      // Block style
+      "style": "consistent"
+    },
+  
+    // MD047/single-trailing-newline : Files should end with a single newline character : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md047.md
+    "MD047": true,
+  
+    // MD048/code-fence-style : Code fence style : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md048.md
+    "MD048": {
+      // Code fence style
+      "style": "consistent"
+    },
+  
+    // MD049/emphasis-style : Emphasis style : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md049.md
+    "MD049": {
+      // Emphasis style
+      "style": "consistent"
+    },
+  
+    // MD050/strong-style : Strong style : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md050.md
+    "MD050": {
+      // Strong style
+      "style": "consistent"
+    },
+  
+    // MD051/link-fragments : Link fragments should be valid : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md051.md
+    "MD051": {
+      // Ignore case of fragments
+      "ignore_case": false
+    },
+  
+    // MD052/reference-links-images : Reference links and images should use a label that is defined : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md052.md
+    "MD052": {
+      // Include shortcut syntax
+      "shortcut_syntax": false
+    },
+  
+    // MD053/link-image-reference-definitions : Link and image reference definitions should be needed : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md053.md
+    "MD053": {
+      // Ignored definitions
+      "ignored_definitions": [
+        "//"
+      ]
+    },
+  
+    // MD054/link-image-style : Link and image style : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md054.md
+    "MD054": {
+      // Allow autolinks
+      "autolink": true,
+      // Allow inline links and images
+      "inline": true,
+      // Allow full reference links and images
+      "full": true,
+      // Allow collapsed reference links and images
+      "collapsed": true,
+      // Allow shortcut reference links and images
+      "shortcut": true,
+      // Allow URLs as inline links
+      "url_inline": true
+    },
+  
+    // MD055/table-pipe-style : Table pipe style : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md055.md
+    "MD055": {
+      // Table pipe style
+      "style": "consistent"
+    },
+  
+    // MD056/table-column-count : Table column count : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md056.md
+    "MD056": true,
+  
+    // MD058/blanks-around-tables : Tables should be surrounded by blank lines : https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md058.md
+    "MD058": true
+  }
\ No newline at end of file


Test patch: