Repository: gohugoio/hugo (language: go)

Issue:
Make `.RenderString` render shortcodes
As [requested](https://github.com/gohugoio/hugo/issues/6666#issuecomment-568779670) I'm re-creating this issue.

Now that we have `.RenderString`. The problem mentioned in https://github.com/gohugoio/hugo/issues/6094#issuecomment-513116608 has been lifted and we should consider having `.RenderString` process shortcodes.

### Do we really need this?

When building a page with blocks of data (CMS Blocks,  Page Builders and so on), most of those are saved as Front Matter. It then falls on the developer to range on those blocks and apply the proper data processing/partial loading etc...

In the context of a very nice to have page builder, Hugo shortcodes are currently unusable by the editor and any search-and-replace workaround cooked up by developers (been there) will never match the reliability and usability of Hugo's shortcode API.

### Will it break code?

The `.RenderString` method is fairly new and I don't see anyone including shortcodes in there with the expectation that they would not be processed. So I think: No


Discussion / hints:
I just came across `.RenderString` as a way to render some markdown that's in the front matter of a page and was surprised it didn't handle shortcodes (then I read the docs and saw it stated).

I see value in it handling that, if not, an alternative like `shortcodify` like was mentioned in the parent issue.
also seeing a need for this..
Filling up my notification screen with "I need this, too" does not help.
Hello @mlake, usually a "thumb up" emoji on the issue's description is enough to notify of your interest in this issue being treated. You'll get the same effect without the noise. Thanks a ton!
This is a huge need for us. Partials allow for a lot of DRYness in our docs site and not being able to include markdown with other shortcodes in it (like relrefs) is painful.
Hi I stumbled on this issue, and I wonder if the _new_ `.RenderString` that could process shortcodes could help me in the below case.

----

I have a mardown table that I'd like to wrap in a div, and I'd have shortcodes to highlight / mark some text in this table.

```
{{< wrapTable >}}

| Memory Type | # of Segments | Used Memory(bytes) | Used Memory(%) | Free Memory(bytes) | Free Memory(%) | Total Memory(bytes) |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Internal | 13 | 1 191 740 | {{< mark >}}**88,4**{{< /mark >}} | 156 452 | 11,6 | 1 348 192 |
| Object (reversed) | 1 | 402 653 184 | {{< mark >}}**100**{{< /mark >}} | 0 | 0 | {{< mark >}}**402 653 184**{{< /mark >}} |
| Class | 9 735 | 228 637 344 | {{< mark >}}**90,96**{{< /mark >}} | 22 724 116 | 9,04 | {{< mark >}}**251 361 460**{{< /mark >}} |
| JIT Code Cache | 5 | 41 943 040 | {{< mark >}}**100**{{< /mark >}} | 0 | 0 | 41 943 040 |
| JIT Data Cache | 3 | 17 018 496 | 67,63 | 8 147 328 | 32,37 | 25 165 824 |
| Overall | 9 757 | 691 443 804 | 95,71 | 31 027 896 | 4,29 | 722 471 700 |

{{< /wrapTable >}}
```

My shortcodes are 

* `wrapTable.html`
```
<div class="table-wrapper">
    <!-- {{ .Inner | markdownify }} -->
</div>
```
* `mark.html`
```
<mark>{{ .Inner | markdownify }}</mark>
```

However with this code the table is broken as soon as there's the `{{< mark >}}` shortcode.

I've tried using `$.Page.RenderString`  instead of `mardownify` but this didn't actually change anything. So for now I'm using a hacky way with two shortcodes for the table wrapper.

* `openWrapTable` that output the `<div>`
* `closeWrapTable` that output `</div>`

Found this; my use case (which I've not yet seen mentioned in any of the various issue discussions so far, apologies if I missed it) is shortcodes with [Remark.js](https://github.com/gnab/remark). Typical implementation uses `.RawContent`, which of course means shortcodes go un-rendered. Being able to do `.RawContent | shortcodify`, or `.RawContent | shortcodify .` to [have the `.Page` context](https://github.com/gohugoio/hugo/issues/6094) if necessary would be excellent.
I'm really missing this feature. I need to process youtube links and "ref" links from frontmatter blocks. Now I'm using really wild regex...
~~Hmmm....this seems to be working in `v0.81.0+extended`.~~

Never mind...I was looking at the wrong case in my tests.
> ~Hmmm....this seems to be working in `v0.81.0+extended`.~
> 
> Never mind...I was looking at the wrong case in my tests.

Not working, confirmed on v0.82.0
Test case:

```
git clone --single-branch -b hugo-github-issue-6703 https://github.com/jmooring/hugo-testing hugo-github-issue-6703
cd hugo-github-issue-6703
hugo server
```
I'm working on the next generation of MIT Open CourseWare and we're using Hugo to build course websites.  This functionality would be very useful for us as we use a JSON data template to store a lot of information about the course.  We have at least one field in there right now where we store markdown text and we need to be able to process shortcodes through either `markdownify` or `RenderString`.
Another place I've noticed that this being a problem recently is when you have an outer and inner shortcode and the outer shortcode is being called with the `%` delimiters.  For example:

`outer.html`:
```
<div>
{{ .Inner }}
</div>
```

`mid.html`:
```
<p>
{{ .Inner }}
</p>
```

`inner.html`:
```
{{ .Get 0 }}
```

`test.md`:
```
{{< outer >}}
{{% mid %}}
{{< inner "test" >}}
{{% /mid %}}
{{< outer />}}
```

In this example, in your output you would get:

```
<div>
<p>
</p>
</div>
```

Fix patch:
diff --git a/common/text/transform.go b/common/text/transform.go
index b324b54c12f..de093af0d4c 100644
--- a/common/text/transform.go
+++ b/common/text/transform.go
@@ -61,3 +61,18 @@ func Puts(s string) string {
 	}
 	return s + "\n"
 }
+
+// VisitLinesAfter calls the given function for each line, including newlines, in the given string.
+func VisitLinesAfter(s string, fn func(line string)) {
+	high := strings.IndexRune(s, '\n')
+	for high != -1 {
+		fn(s[:high+1])
+		s = s[high+1:]
+
+		high = strings.IndexRune(s, '\n')
+	}
+
+	if s != "" {
+		fn(s)
+	}
+}
diff --git a/hugolib/content_map_page.go b/hugolib/content_map_page.go
index a16e4720d9d..c6522809efe 100644
--- a/hugolib/content_map_page.go
+++ b/hugolib/content_map_page.go
@@ -163,7 +163,7 @@ func (m *pageMap) newPageFromContentNode(n *contentNode, parentBucket *pagesMapB
 		},
 	}
 
-	ps.shortcodeState = newShortcodeHandler(ps, ps.s, nil)
+	ps.shortcodeState = newShortcodeHandler(ps, ps.s)
 
 	if err := ps.mapContent(parentBucket, metaProvider); err != nil {
 		return nil, ps.wrapError(err)
diff --git a/hugolib/page.go b/hugolib/page.go
index 8bf13e32067..e37b47300c4 100644
--- a/hugolib/page.go
+++ b/hugolib/page.go
@@ -336,7 +336,7 @@ func (p *pageState) HasShortcode(name string) bool {
 		return false
 	}
 
-	return p.shortcodeState.nameSet[name]
+	return p.shortcodeState.hasName(name)
 }
 
 func (p *pageState) Site() page.Site {
@@ -610,13 +610,30 @@ func (p *pageState) getContentConverter() converter.Converter {
 }
 
 func (p *pageState) mapContent(bucket *pagesMapBucket, meta *pageMeta) error {
-	s := p.shortcodeState
-
-	rn := &pageContentMap{
+	p.cmap = &pageContentMap{
 		items: make([]any, 0, 20),
 	}
 
-	iter := p.source.parsed.Iterator()
+	return p.mapContentForResult(
+		p.source.parsed,
+		p.shortcodeState,
+		p.cmap,
+		meta.markup,
+		func(m map[string]interface{}) error {
+			return meta.setMetadata(bucket, p, m)
+		},
+	)
+}
+
+func (p *pageState) mapContentForResult(
+	result pageparser.Result,
+	s *shortcodeHandler,
+	rn *pageContentMap,
+	markup string,
+	withFrontMatter func(map[string]any) error,
+) error {
+
+	iter := result.Iterator()
 
 	fail := func(err error, i pageparser.Item) error {
 		if fe, ok := err.(herrors.FileError); ok {
@@ -660,8 +677,10 @@ Loop:
 				}
 			}
 
-			if err := meta.setMetadata(bucket, p, m); err != nil {
-				return err
+			if withFrontMatter != nil {
+				if err := withFrontMatter(m); err != nil {
+					return err
+				}
 			}
 
 			frontMatterSet = true
@@ -697,7 +716,7 @@ Loop:
 			p.source.posBodyStart = posBody
 			p.source.hasSummaryDivider = true
 
-			if meta.markup != "html" {
+			if markup != "html" {
 				// The content will be rendered by Goldmark or similar,
 				// and we need to track the summary.
 				rn.AddReplacement(internalSummaryDividerPre, it)
@@ -720,7 +739,7 @@ Loop:
 			}
 
 			if currShortcode.name != "" {
-				s.nameSet[currShortcode.name] = true
+				s.addName(currShortcode.name)
 			}
 
 			if currShortcode.params == nil {
@@ -752,16 +771,14 @@ Loop:
 		}
 	}
 
-	if !frontMatterSet {
+	if !frontMatterSet && withFrontMatter != nil {
 		// Page content without front matter. Assign default front matter from
 		// cascades etc.
-		if err := meta.setMetadata(bucket, p, nil); err != nil {
+		if err := withFrontMatter(nil); err != nil {
 			return err
 		}
 	}
 
-	p.cmap = rn
-
 	return nil
 }
 
diff --git a/hugolib/page__content.go b/hugolib/page__content.go
index f6f4564ebf0..5871884545d 100644
--- a/hugolib/page__content.go
+++ b/hugolib/page__content.go
@@ -39,12 +39,12 @@ type pageContent struct {
 }
 
 // returns the content to be processed by Goldmark or similar.
-func (p pageContent) contentToRender(renderedShortcodes map[string]string) []byte {
-	source := p.source.parsed.Input()
+func (p pageContent) contentToRender(parsed pageparser.Result, pm *pageContentMap, renderedShortcodes map[string]string) []byte {
+	source := parsed.Input()
 
 	c := make([]byte, 0, len(source)+(len(source)/10))
 
-	for _, it := range p.cmap.items {
+	for _, it := range pm.items {
 		switch v := it.(type) {
 		case pageparser.Item:
 			c = append(c, source[v.Pos:v.Pos+len(v.Val)]...)
diff --git a/hugolib/page__per_output.go b/hugolib/page__per_output.go
index 3d0c6ad1d70..2bf16dd9ee6 100644
--- a/hugolib/page__per_output.go
+++ b/hugolib/page__per_output.go
@@ -25,9 +25,11 @@ import (
 
 	"errors"
 
+	"github.com/gohugoio/hugo/common/herrors"
 	"github.com/gohugoio/hugo/common/text"
 	"github.com/gohugoio/hugo/common/types/hstring"
 	"github.com/gohugoio/hugo/identity"
+	"github.com/gohugoio/hugo/parser/pageparser"
 	"github.com/mitchellh/mapstructure"
 	"github.com/spf13/cast"
 
@@ -117,7 +119,7 @@ func newPageContentOutput(p *pageState, po *pageOutput) (*pageContentOutput, err
 			p.pageOutputTemplateVariationsState.Store(2)
 		}
 
-		cp.workContent = p.contentToRender(cp.contentPlaceholders)
+		cp.workContent = p.contentToRender(p.source.parsed, p.cmap, cp.contentPlaceholders)
 
 		isHTML := cp.p.m.markup == "html"
 
@@ -332,11 +334,12 @@ func (p *pageContentOutput) WordCount() int {
 }
 
 func (p *pageContentOutput) RenderString(args ...any) (template.HTML, error) {
+	defer herrors.Recover()
 	if len(args) < 1 || len(args) > 2 {
 		return "", errors.New("want 1 or 2 arguments")
 	}
 
-	var s string
+	var contentToRender string
 	opts := defaultRenderStringOpts
 	sidx := 1
 
@@ -353,16 +356,16 @@ func (p *pageContentOutput) RenderString(args ...any) (template.HTML, error) {
 		}
 	}
 
-	contentToRender := args[sidx]
+	contentToRenderv := args[sidx]
 
-	if _, ok := contentToRender.(hstring.RenderedString); ok {
+	if _, ok := contentToRenderv.(hstring.RenderedString); ok {
 		// This content is already rendered, this is potentially
 		// a infinite recursion.
 		return "", errors.New("text is already rendered, repeating it may cause infinite recursion")
 	}
 
 	var err error
-	s, err = cast.ToStringE(contentToRender)
+	contentToRender, err = cast.ToStringE(contentToRenderv)
 	if err != nil {
 		return "", err
 	}
@@ -381,20 +384,79 @@ func (p *pageContentOutput) RenderString(args ...any) (template.HTML, error) {
 		}
 	}
 
-	c, err := p.renderContentWithConverter(conv, []byte(s), false)
-	if err != nil {
-		return "", p.p.wrapError(err)
-	}
+	var rendered []byte
+
+	if strings.Contains(contentToRender, "{{") {
+		// Probably a shortcode.
+		parsed, err := pageparser.ParseMain(strings.NewReader(contentToRender), pageparser.Config{})
+		if err != nil {
+			return "", err
+		}
+		pm := &pageContentMap{
+			items: make([]any, 0, 20),
+		}
+		s := newShortcodeHandler(p.p, p.p.s)
+
+		if err := p.p.mapContentForResult(
+			parsed,
+			s,
+			pm,
+			opts.Markup,
+			nil,
+		); err != nil {
+			return "", err
+		}
+
+		placeholders, hasShortcodeVariants, err := s.renderShortcodesForPage(p.p, p.f)
+		if err != nil {
+			return "", err
+		}
+
+		if hasShortcodeVariants {
+			p.p.pageOutputTemplateVariationsState.Store(2)
+		}
+
+		b, err := p.renderContentWithConverter(conv, p.p.contentToRender(parsed, pm, placeholders), false)
+		if err != nil {
+			return "", p.p.wrapError(err)
+		}
+		rendered = b.Bytes()
 
-	b := c.Bytes()
+		if p.placeholdersEnabled {
+			// ToC was accessed via .Page.TableOfContents in the shortcode,
+			// at a time when the ToC wasn't ready.
+			if _, err := p.p.Content(); err != nil {
+				return "", err
+			}
+			placeholders[tocShortcodePlaceholder] = string(p.tableOfContents)
+		}
+
+		if pm.hasNonMarkdownShortcode || p.placeholdersEnabled {
+			rendered, err = replaceShortcodeTokens(rendered, placeholders)
+			if err != nil {
+				return "", err
+			}
+		}
+
+		// We need a consolidated view in $page.HasShortcode
+		p.p.shortcodeState.transferNames(s)
+
+	} else {
+		c, err := p.renderContentWithConverter(conv, []byte(contentToRender), false)
+		if err != nil {
+			return "", p.p.wrapError(err)
+		}
+
+		rendered = c.Bytes()
+	}
 
 	if opts.Display == "inline" {
 		// We may have to rethink this in the future when we get other
 		// renderers.
-		b = p.p.s.ContentSpec.TrimShortHTML(b)
+		rendered = p.p.s.ContentSpec.TrimShortHTML(rendered)
 	}
 
-	return template.HTML(string(b)), nil
+	return template.HTML(string(rendered)), nil
 }
 
 func (p *pageContentOutput) RenderWithTemplateInfo(info tpl.Info, layout ...string) (template.HTML, error) {
diff --git a/hugolib/shortcode.go b/hugolib/shortcode.go
index e718d861ac8..366875b88a5 100644
--- a/hugolib/shortcode.go
+++ b/hugolib/shortcode.go
@@ -170,6 +170,8 @@ type shortcode struct {
 	ordinal   int
 	err       error
 
+	indentation string // indentation from source.
+
 	info   tpl.Info       // One of the output formats (arbitrary)
 	templs []tpl.Template // All output formats
 
@@ -248,13 +250,14 @@ type shortcodeHandler struct {
 	shortcodes []*shortcode
 
 	// All the shortcode names in this set.
-	nameSet map[string]bool
+	nameSet   map[string]bool
+	nameSetMu sync.RWMutex
 
 	// Configuration
 	enableInlineShortcodes bool
 }
 
-func newShortcodeHandler(p *pageState, s *Site, placeholderFunc func() string) *shortcodeHandler {
+func newShortcodeHandler(p *pageState, s *Site) *shortcodeHandler {
 	sh := &shortcodeHandler{
 		p:                      p,
 		s:                      s,
@@ -398,6 +401,22 @@ func renderShortcode(
 		return "", false, fe
 	}
 
+	if len(sc.inner) == 0 && len(sc.indentation) > 0 {
+		b := bp.GetBuffer()
+		i := 0
+		text.VisitLinesAfter(result, func(line string) {
+			// The first line is correctly indented.
+			if i > 0 {
+				b.WriteString(sc.indentation)
+			}
+			i++
+			b.WriteString(line)
+		})
+
+		result = b.String()
+		bp.PutBuffer(b)
+	}
+
 	return result, hasVariants, err
 }
 
@@ -405,6 +424,28 @@ func (s *shortcodeHandler) hasShortcodes() bool {
 	return s != nil && len(s.shortcodes) > 0
 }
 
+func (s *shortcodeHandler) addName(name string) {
+	s.nameSetMu.Lock()
+	defer s.nameSetMu.Unlock()
+	s.nameSet[name] = true
+}
+
+func (s *shortcodeHandler) transferNames(in *shortcodeHandler) {
+	s.nameSetMu.Lock()
+	defer s.nameSetMu.Unlock()
+	for k := range in.nameSet {
+		s.nameSet[k] = true
+	}
+
+}
+
+func (s *shortcodeHandler) hasName(name string) bool {
+	s.nameSetMu.RLock()
+	defer s.nameSetMu.RUnlock()
+	_, ok := s.nameSet[name]
+	return ok
+}
+
 func (s *shortcodeHandler) renderShortcodesForPage(p *pageState, f output.Format) (map[string]string, bool, error) {
 	rendered := make(map[string]string)
 
@@ -447,6 +488,15 @@ func (s *shortcodeHandler) extractShortcode(ordinal, level int, pt *pageparser.I
 	}
 	sc := &shortcode{ordinal: ordinal}
 
+	// Back up one to identify any indentation.
+	if pt.Pos() > 0 {
+		pt.Backup()
+		item := pt.Next()
+		if item.IsIndentation() {
+			sc.indentation = string(item.Val)
+		}
+	}
+
 	cnt := 0
 	nestedOrdinal := 0
 	nextLevel := level + 1
@@ -476,7 +526,7 @@ Loop:
 				nested, err := s.extractShortcode(nestedOrdinal, nextLevel, pt)
 				nestedOrdinal++
 				if nested != nil && nested.name != "" {
-					s.nameSet[nested.name] = true
+					s.addName(nested.name)
 				}
 
 				if err == nil {
diff --git a/media/mediaType.go b/media/mediaType.go
index 69bb9182acd..3ac3123ac81 100644
--- a/media/mediaType.go
+++ b/media/mediaType.go
@@ -257,7 +257,8 @@ var (
 	OpenTypeFontType = newMediaType("font", "otf", []string{"otf"})
 
 	// Common document types
-	PDFType = newMediaType("application", "pdf", []string{"pdf"})
+	PDFType      = newMediaType("application", "pdf", []string{"pdf"})
+	MarkdownType = newMediaType("text", "markdown", []string{"md", "markdown"})
 
 	// Common video types
 	AVIType  = newMediaType("video", "x-msvideo", []string{"avi"})
@@ -278,6 +279,7 @@ var DefaultTypes = Types{
 	SCSSType,
 	SASSType,
 	HTMLType,
+	MarkdownType,
 	JavascriptType,
 	TypeScriptType,
 	TSXType,
diff --git a/output/outputFormat.go b/output/outputFormat.go
index 1f1556eafe4..722079df9a7 100644
--- a/output/outputFormat.go
+++ b/output/outputFormat.go
@@ -133,6 +133,14 @@ var (
 		Weight: 10,
 	}
 
+	MarkdownFormat = Format{
+		Name:        "MARKDOWN",
+		MediaType:   media.MarkdownType,
+		BaseName:    "index",
+		Rel:         "alternate",
+		IsPlainText: true,
+	}
+
 	JSONFormat = Format{
 		Name:        "JSON",
 		MediaType:   media.JSONType,
@@ -183,6 +191,7 @@ var DefaultFormats = Formats{
 	CSVFormat,
 	HTMLFormat,
 	JSONFormat,
+	MarkdownFormat,
 	WebAppManifestFormat,
 	RobotsTxtFormat,
 	RSSFormat,
diff --git a/parser/pageparser/item.go b/parser/pageparser/item.go
index f23341ea8a7..52546be4193 100644
--- a/parser/pageparser/item.go
+++ b/parser/pageparser/item.go
@@ -18,6 +18,8 @@ import (
 	"fmt"
 	"regexp"
 	"strconv"
+
+	"github.com/yuin/goldmark/util"
 )
 
 type Item struct {
@@ -64,7 +66,11 @@ func (i Item) ValTyped() any {
 }
 
 func (i Item) IsText() bool {
-	return i.Type == tText
+	return i.Type == tText || i.Type == tIndentation
+}
+
+func (i Item) IsIndentation() bool {
+	return i.Type == tIndentation
 }
 
 func (i Item) IsNonWhitespace() bool {
@@ -125,6 +131,8 @@ func (i Item) String() string {
 		return "EOF"
 	case i.Type == tError:
 		return string(i.Val)
+	case i.Type == tIndentation:
+		return fmt.Sprintf("%s:[%s]", i.Type, util.VisualizeSpaces(i.Val))
 	case i.Type > tKeywordMarker:
 		return fmt.Sprintf("<%s>", i.Val)
 	case len(i.Val) > 50:
@@ -159,6 +167,8 @@ const (
 	tScParam
 	tScParamVal
 
+	tIndentation
+
 	tText // plain text
 
 	// preserved for later - keywords come after this
diff --git a/parser/pageparser/itemtype_string.go b/parser/pageparser/itemtype_string.go
index 632afaecc00..b0b849adecf 100644
--- a/parser/pageparser/itemtype_string.go
+++ b/parser/pageparser/itemtype_string.go
@@ -4,9 +4,36 @@ package pageparser
 
 import "strconv"
 
-const _ItemType_name = "tErrortEOFTypeHTMLStartTypeLeadSummaryDividerTypeFrontMatterYAMLTypeFrontMatterTOMLTypeFrontMatterJSONTypeFrontMatterORGTypeEmojiTypeIgnoretLeftDelimScNoMarkuptRightDelimScNoMarkuptLeftDelimScWithMarkuptRightDelimScWithMarkuptScClosetScNametScNameInlinetScParamtScParamValtTexttKeywordMarker"
+func _() {
+	// An "invalid array index" compiler error signifies that the constant values have changed.
+	// Re-run the stringer command to generate them again.
+	var x [1]struct{}
+	_ = x[tError-0]
+	_ = x[tEOF-1]
+	_ = x[TypeLeadSummaryDivider-2]
+	_ = x[TypeFrontMatterYAML-3]
+	_ = x[TypeFrontMatterTOML-4]
+	_ = x[TypeFrontMatterJSON-5]
+	_ = x[TypeFrontMatterORG-6]
+	_ = x[TypeEmoji-7]
+	_ = x[TypeIgnore-8]
+	_ = x[tLeftDelimScNoMarkup-9]
+	_ = x[tRightDelimScNoMarkup-10]
+	_ = x[tLeftDelimScWithMarkup-11]
+	_ = x[tRightDelimScWithMarkup-12]
+	_ = x[tScClose-13]
+	_ = x[tScName-14]
+	_ = x[tScNameInline-15]
+	_ = x[tScParam-16]
+	_ = x[tScParamVal-17]
+	_ = x[tIndentation-18]
+	_ = x[tText-19]
+	_ = x[tKeywordMarker-20]
+}
+
+const _ItemType_name = "tErrortEOFTypeLeadSummaryDividerTypeFrontMatterYAMLTypeFrontMatterTOMLTypeFrontMatterJSONTypeFrontMatterORGTypeEmojiTypeIgnoretLeftDelimScNoMarkuptRightDelimScNoMarkuptLeftDelimScWithMarkuptRightDelimScWithMarkuptScClosetScNametScNameInlinetScParamtScParamValtIndentationtTexttKeywordMarker"
 
-var _ItemType_index = [...]uint16{0, 6, 10, 23, 45, 64, 83, 102, 120, 129, 139, 159, 180, 202, 225, 233, 240, 253, 261, 272, 277, 291}
+var _ItemType_index = [...]uint16{0, 6, 10, 32, 51, 70, 89, 107, 116, 126, 146, 167, 189, 212, 220, 227, 240, 248, 259, 271, 276, 290}
 
 func (i ItemType) String() string {
 	if i < 0 || i >= ItemType(len(_ItemType_index)-1) {
diff --git a/parser/pageparser/pagelexer.go b/parser/pageparser/pagelexer.go
index 20965ac0e91..770f26eb9dc 100644
--- a/parser/pageparser/pagelexer.go
+++ b/parser/pageparser/pagelexer.go
@@ -120,6 +120,7 @@ func (l *pageLexer) next() rune {
 	runeValue, runeWidth := utf8.DecodeRune(l.input[l.pos:])
 	l.width = runeWidth
 	l.pos += l.width
+
 	return runeValue
 }
 
@@ -137,8 +138,34 @@ func (l *pageLexer) backup() {
 
 // sends an item back to the client.
 func (l *pageLexer) emit(t ItemType) {
+	defer func() {
+		l.start = l.pos
+	}()
+
+	if t == tText {
+		// Identify any trailing whitespace/intendation.
+		// We currently only care about the last one.
+		for i := l.pos - 1; i >= l.start; i-- {
+			b := l.input[i]
+			if b != ' ' && b != '\t' && b != '\r' && b != '\n' {
+				break
+			}
+			if i == l.start && b != '\n' {
+				l.items = append(l.items, Item{tIndentation, l.start, l.input[l.start:l.pos], false})
+				return
+			} else if b == '\n' && i < l.pos-1 {
+				l.items = append(l.items, Item{t, l.start, l.input[l.start : i+1], false})
+				l.items = append(l.items, Item{tIndentation, i + 1, l.input[i+1 : l.pos], false})
+				return
+			} else if b == '\n' && i == l.pos-1 {
+				break
+			}
+
+		}
+	}
+
 	l.items = append(l.items, Item{t, l.start, l.input[l.start:l.pos], false})
-	l.start = l.pos
+
 }
 
 // sends a string item back to the client.
diff --git a/parser/pageparser/pageparser.go b/parser/pageparser/pageparser.go
index bfbedcf2663..67abefc3089 100644
--- a/parser/pageparser/pageparser.go
+++ b/parser/pageparser/pageparser.go
@@ -149,6 +149,11 @@ func (t *Iterator) Backup() {
 	t.lastPos--
 }
 
+// Pos returns the current position in the input.
+func (t *Iterator) Pos() int {
+	return t.lastPos
+}
+
 // check for non-error and non-EOF types coming next
 func (t *Iterator) IsValueNext() bool {
 	i := t.Peek()


Test patch:
diff --git a/common/text/transform_test.go b/common/text/transform_test.go
index 992dd524c80..41447715f47 100644
--- a/common/text/transform_test.go
+++ b/common/text/transform_test.go
@@ -41,3 +41,36 @@ func TestPuts(t *testing.T) {
 	c.Assert(Puts("\nA\n"), qt.Equals, "\nA\n")
 	c.Assert(Puts(""), qt.Equals, "")
 }
+
+func TestVisitLinesAfter(t *testing.T) {
+	const lines = `line 1
+line 2
+
+line 3`
+
+	var collected []string
+
+	VisitLinesAfter(lines, func(s string) {
+		collected = append(collected, s)
+	})
+
+	c := qt.New(t)
+
+	c.Assert(collected, qt.DeepEquals, []string{"line 1\n", "line 2\n", "\n", "line 3"})
+
+}
+
+func BenchmarkVisitLinesAfter(b *testing.B) {
+	const lines = `line 1
+	line 2
+	
+	line 3`
+
+	for i := 0; i < b.N; i++ {
+		VisitLinesAfter(lines, func(s string) {
+
+		})
+
+	}
+
+}
diff --git a/hugolib/content_render_hooks_test.go b/hugolib/content_render_hooks_test.go
index 33ebe1f41d9..dbfd464590c 100644
--- a/hugolib/content_render_hooks_test.go
+++ b/hugolib/content_render_hooks_test.go
@@ -18,7 +18,6 @@ import (
 	"testing"
 
 	qt "github.com/frankban/quicktest"
-	"github.com/gohugoio/hugo/common/loggers"
 )
 
 func TestRenderHookEditNestedPartial(t *testing.T) {
@@ -428,72 +427,3 @@ Image:
 <p>html-image: image.jpg|Text: Hello<br> Goodbye|Plain: Hello GoodbyeEND</p>
 `)
 }
-
-func TestRenderString(t *testing.T) {
-	b := newTestSitesBuilder(t)
-
-	b.WithTemplates("index.html", `
-{{ $p := site.GetPage "p1.md" }}
-{{ $optBlock := dict "display" "block" }}
-{{ $optOrg := dict "markup" "org" }}
-RSTART:{{ "**Bold Markdown**" | $p.RenderString }}:REND
-RSTART:{{  "**Bold Block Markdown**" | $p.RenderString  $optBlock }}:REND
-RSTART:{{  "/italic org mode/" | $p.RenderString  $optOrg }}:REND
-RSTART:{{ "## Header2" | $p.RenderString }}:REND
-
-
-`, "_default/_markup/render-heading.html", "Hook Heading: {{ .Level }}")
-
-	b.WithContent("p1.md", `---
-title: "p1"
----
-`,
-	)
-
-	b.Build(BuildCfg{})
-
-	b.AssertFileContent("public/index.html", `
-RSTART:<strong>Bold Markdown</strong>:REND
-RSTART:<p><strong>Bold Block Markdown</strong></p>
-RSTART:<em>italic org mode</em>:REND
-RSTART:Hook Heading: 2:REND
-`)
-}
-
-// https://github.com/gohugoio/hugo/issues/6882
-func TestRenderStringOnListPage(t *testing.T) {
-	renderStringTempl := `
-{{ .RenderString "**Hello**" }}
-`
-	b := newTestSitesBuilder(t)
-	b.WithContent("mysection/p1.md", `FOO`)
-	b.WithTemplates(
-		"index.html", renderStringTempl,
-		"_default/list.html", renderStringTempl,
-		"_default/single.html", renderStringTempl,
-	)
-
-	b.Build(BuildCfg{})
-
-	for _, filename := range []string{
-		"index.html",
-		"mysection/index.html",
-		"categories/index.html",
-		"tags/index.html",
-		"mysection/p1/index.html",
-	} {
-		b.AssertFileContent("public/"+filename, `<strong>Hello</strong>`)
-	}
-}
-
-// Issue 9433
-func TestRenderStringOnPageNotBackedByAFile(t *testing.T) {
-	t.Parallel()
-	logger := loggers.NewWarningLogger()
-	b := newTestSitesBuilder(t).WithLogger(logger).WithConfigFile("toml", `
-disableKinds = ["page", "section", "taxonomy", "term"]	
-`)
-	b.WithTemplates("index.html", `{{ .RenderString "**Hello**" }}`).WithContent("p1.md", "")
-	b.BuildE(BuildCfg{})
-	b.Assert(int(logger.LogCounters().WarnCounter.Count()), qt.Equals, 0)
-}
diff --git a/hugolib/renderstring_test.go b/hugolib/renderstring_test.go
new file mode 100644
index 00000000000..d2f453c3303
--- /dev/null
+++ b/hugolib/renderstring_test.go
@@ -0,0 +1,162 @@
+// Copyright 2022 The Hugo Authors. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless requiredF by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package hugolib
+
+import (
+	"testing"
+
+	qt "github.com/frankban/quicktest"
+	"github.com/gohugoio/hugo/common/loggers"
+)
+
+func TestRenderString(t *testing.T) {
+	b := newTestSitesBuilder(t)
+
+	b.WithTemplates("index.html", `
+{{ $p := site.GetPage "p1.md" }}
+{{ $optBlock := dict "display" "block" }}
+{{ $optOrg := dict "markup" "org" }}
+RSTART:{{ "**Bold Markdown**" | $p.RenderString }}:REND
+RSTART:{{  "**Bold Block Markdown**" | $p.RenderString  $optBlock }}:REND
+RSTART:{{  "/italic org mode/" | $p.RenderString  $optOrg }}:REND
+RSTART:{{ "## Header2" | $p.RenderString }}:REND
+
+
+`, "_default/_markup/render-heading.html", "Hook Heading: {{ .Level }}")
+
+	b.WithContent("p1.md", `---
+title: "p1"
+---
+`,
+	)
+
+	b.Build(BuildCfg{})
+
+	b.AssertFileContent("public/index.html", `
+RSTART:<strong>Bold Markdown</strong>:REND
+RSTART:<p><strong>Bold Block Markdown</strong></p>
+RSTART:<em>italic org mode</em>:REND
+RSTART:Hook Heading: 2:REND
+`)
+}
+
+// https://github.com/gohugoio/hugo/issues/6882
+func TestRenderStringOnListPage(t *testing.T) {
+	renderStringTempl := `
+{{ .RenderString "**Hello**" }}
+`
+	b := newTestSitesBuilder(t)
+	b.WithContent("mysection/p1.md", `FOO`)
+	b.WithTemplates(
+		"index.html", renderStringTempl,
+		"_default/list.html", renderStringTempl,
+		"_default/single.html", renderStringTempl,
+	)
+
+	b.Build(BuildCfg{})
+
+	for _, filename := range []string{
+		"index.html",
+		"mysection/index.html",
+		"categories/index.html",
+		"tags/index.html",
+		"mysection/p1/index.html",
+	} {
+		b.AssertFileContent("public/"+filename, `<strong>Hello</strong>`)
+	}
+}
+
+// Issue 9433
+func TestRenderStringOnPageNotBackedByAFile(t *testing.T) {
+	t.Parallel()
+	logger := loggers.NewWarningLogger()
+	b := newTestSitesBuilder(t).WithLogger(logger).WithConfigFile("toml", `
+disableKinds = ["page", "section", "taxonomy", "term"]	
+`)
+	b.WithTemplates("index.html", `{{ .RenderString "**Hello**" }}`).WithContent("p1.md", "")
+	b.BuildE(BuildCfg{})
+	b.Assert(int(logger.LogCounters().WarnCounter.Count()), qt.Equals, 0)
+}
+
+func TestRenderStringWithShortcode(t *testing.T) {
+	t.Parallel()
+
+	filesTemplate := `
+-- config.toml --
+title = "Hugo Rocks!"
+enableInlineShortcodes = true
+-- content/p1/index.md --
+---
+title: "P1"
+---
+## First
+-- layouts/shortcodes/mark1.md --
+{{ .Inner }}
+-- layouts/shortcodes/mark2.md --
+1. Item Mark2 1
+1. Item Mark2 2
+   1. Item Mark2 2-1
+1. Item Mark2 3
+-- layouts/shortcodes/myhthml.html --
+Title: {{ .Page.Title }}
+TableOfContents: {{ .Page.TableOfContents }}
+Page Type: {{ printf "%T" .Page }}
+-- layouts/_default/single.html --
+{{ .RenderString "Markdown: {{% mark2 %}}|HTML: {{< myhthml >}}|Inline: {{< foo.inline >}}{{ site.Title }}{{< /foo.inline >}}|" }}
+HasShortcode: mark2:{{ .HasShortcode "mark2" }}:true
+HasShortcode: foo:{{ .HasShortcode "foo" }}:false
+
+`
+
+	t.Run("Basic", func(t *testing.T) {
+
+		b := NewIntegrationTestBuilder(
+			IntegrationTestConfig{
+				T:           t,
+				TxtarString: filesTemplate,
+			},
+		).Build()
+
+		b.AssertFileContent("public/p1/index.html",
+			"<p>Markdown: 1. Item Mark2 1</p>\n<ol>\n<li>Item Mark2 2\n<ol>\n<li>Item Mark2 2-1</li>\n</ol>\n</li>\n<li>Item Mark2 3|",
+			"<a href=\"#first\">First</a>", // ToC
+			`
+HTML: Title: P1
+Inline: Hugo Rocks!
+HasShortcode: mark2:true:true
+HasShortcode: foo:false:false
+Page Type: *hugolib.pageForShortcode`,
+		)
+
+	})
+
+	t.Run("Edit shortcode", func(t *testing.T) {
+
+		b := NewIntegrationTestBuilder(
+			IntegrationTestConfig{
+				T:           t,
+				TxtarString: filesTemplate,
+				Running:     true,
+			},
+		).Build()
+
+		b.EditFiles("layouts/shortcodes/myhthml.html", "Edit shortcode").Build()
+
+		b.AssertFileContent("public/p1/index.html",
+			`Edit shortcode`,
+		)
+
+	})
+
+}
diff --git a/hugolib/shortcode_test.go b/hugolib/shortcode_test.go
index 7926a820392..15c27a42e01 100644
--- a/hugolib/shortcode_test.go
+++ b/hugolib/shortcode_test.go
@@ -107,15 +107,9 @@ title: "Shortcodes Galore!"
 			t.Parallel()
 			c := qt.New(t)
 
-			counter := 0
-			placeholderFunc := func() string {
-				counter++
-				return fmt.Sprintf("HAHA%s-%dHBHB", shortcodePlaceholderPrefix, counter)
-			}
-
 			p, err := pageparser.ParseMain(strings.NewReader(test.input), pageparser.Config{})
 			c.Assert(err, qt.IsNil)
-			handler := newShortcodeHandler(nil, s, placeholderFunc)
+			handler := newShortcodeHandler(nil, s)
 			iter := p.Iterator()
 
 			short, err := handler.extractShortcode(0, 0, iter)
@@ -909,3 +903,109 @@ outputs: ["html", "css", "csv", "json"]
 
 	}
 }
+
+// #9821
+func TestShortcodeMarkdownOutputFormat(t *testing.T) {
+	t.Parallel()
+
+	files := `
+-- config.toml --
+-- content/p1.md --
+---
+title: "p1"
+---
+{{< foo >}}
+-- layouts/shortcodes/foo.md --
+§§§
+<x
+§§§
+-- layouts/_default/single.html --
+{{ .Content }}
+`
+
+	b := NewIntegrationTestBuilder(
+		IntegrationTestConfig{
+			T:           t,
+			TxtarString: files,
+			Running:     true,
+		},
+	).Build()
+
+	b.AssertFileContent("public/p1/index.html", `
+<x
+	`)
+
+}
+
+func TestShortcodePreserveIndentation(t *testing.T) {
+	t.Parallel()
+
+	files := `
+-- config.toml --
+-- content/p1.md --
+---
+title: "p1"
+---
+
+## List With Indented Shortcodes
+
+1. List 1
+    {{% mark1 %}}
+	1. Item Mark1 1
+	1. Item Mark1 2
+	{{% mark2 %}}
+	{{% /mark1 %}}
+-- layouts/shortcodes/mark1.md --
+{{ .Inner }}
+-- layouts/shortcodes/mark2.md --
+1. Item Mark2 1
+1. Item Mark2 2
+   1. Item Mark2 2-1
+1. Item Mark2 3
+-- layouts/_default/single.html --
+{{ .Content }}
+`
+
+	b := NewIntegrationTestBuilder(
+		IntegrationTestConfig{
+			T:           t,
+			TxtarString: files,
+			Running:     true,
+		},
+	).Build()
+
+	b.AssertFileContent("public/p1/index.html", "<ol>\n<li>\n<p>List 1</p>\n<ol>\n<li>Item Mark1 1</li>\n<li>Item Mark1 2</li>\n<li>Item Mark2 1</li>\n<li>Item Mark2 2\n<ol>\n<li>Item Mark2 2-1</li>\n</ol>\n</li>\n<li>Item Mark2 3</li>\n</ol>\n</li>\n</ol>")
+
+}
+
+func TestShortcodeCodeblockIndent(t *testing.T) {
+	t.Parallel()
+
+	files := `
+-- config.toml --
+-- content/p1.md --
+---
+title: "p1"
+---
+
+## Code block
+
+    {{% code %}}
+
+-- layouts/shortcodes/code.md --
+echo "foo";
+-- layouts/_default/single.html --
+{{ .Content }}
+`
+
+	b := NewIntegrationTestBuilder(
+		IntegrationTestConfig{
+			T:           t,
+			TxtarString: files,
+			Running:     true,
+		},
+	).Build()
+
+	b.AssertFileContent("public/p1/index.html", "<pre><code>echo &quot;foo&quot;;\n</code></pre>")
+
+}
diff --git a/media/mediaType_test.go b/media/mediaType_test.go
index 34571f7ee00..af7123cb5cd 100644
--- a/media/mediaType_test.go
+++ b/media/mediaType_test.go
@@ -63,7 +63,7 @@ func TestDefaultTypes(t *testing.T) {
 
 	}
 
-	c.Assert(len(DefaultTypes), qt.Equals, 33)
+	c.Assert(len(DefaultTypes), qt.Equals, 34)
 }
 
 func TestGetByType(t *testing.T) {
diff --git a/output/outputFormat_test.go b/output/outputFormat_test.go
index a150dbe7ca4..c5c4534bfd1 100644
--- a/output/outputFormat_test.go
+++ b/output/outputFormat_test.go
@@ -68,7 +68,7 @@ func TestDefaultTypes(t *testing.T) {
 	c.Assert(RSSFormat.NoUgly, qt.Equals, true)
 	c.Assert(CalendarFormat.IsHTML, qt.Equals, false)
 
-	c.Assert(len(DefaultFormats), qt.Equals, 10)
+	c.Assert(len(DefaultFormats), qt.Equals, 11)
 
 }
 
diff --git a/parser/pageparser/pageparser_shortcode_test.go b/parser/pageparser/pageparser_shortcode_test.go
index 54580217ca6..ce12975737d 100644
--- a/parser/pageparser/pageparser_shortcode_test.go
+++ b/parser/pageparser/pageparser_shortcode_test.go
@@ -51,6 +51,9 @@ var shortCodeLexerTests = []lexerTest{
 
 	{"simple with markup", `{{% sc1 %}}`, []Item{tstLeftMD, tstSC1, tstRightMD, tstEOF}},
 	{"with spaces", `{{<     sc1     >}}`, []Item{tstLeftNoMD, tstSC1, tstRightNoMD, tstEOF}},
+	{"indented on new line", "Hello\n    {{% sc1 %}}", []Item{nti(tText, "Hello\n"), nti(tIndentation, "    "), tstLeftMD, tstSC1, tstRightMD, tstEOF}},
+	{"indented on new line tab", "Hello\n\t{{% sc1 %}}", []Item{nti(tText, "Hello\n"), nti(tIndentation, "\t"), tstLeftMD, tstSC1, tstRightMD, tstEOF}},
+	{"indented on first line", "    {{% sc1 %}}", []Item{nti(tIndentation, "    "), tstLeftMD, tstSC1, tstRightMD, tstEOF}},
 	{"mismatched rightDelim", `{{< sc1 %}}`, []Item{
 		tstLeftNoMD, tstSC1,
 		nti(tError, "unrecognized character in shortcode action: U+0025 '%'. Note: Parameters with non-alphanumeric args must be quoted"),