Repository: kovetskiy/mark (go)

Issue:
Handle semantic line breaks when converting markdown file to an HTML file
Thanks for the work on `mark,` this is an amazing tool! It works great for my needs except for one subtle detail.
I believe it's the "opposite" of another issue: https://github.com/kovetskiy/mark/issues/61

Having this markdown snippet which follows the [semantic line breaks](https://sembr.org/) guide:

```markdown
If you want to save the file,
you can click Save.
```

Will generate this HTML file (which I see by using the `--dry-run` flag):

```html
<p>If you want to save the file,
you can click Save.</p>
```

When this HTML gets rendered in Confluence, the line breaks within the `<p>` tag are respected resulting in awkward breaks:

![image](https://user-images.githubusercontent.com/59651540/125070575-110f1880-e0b0-11eb-82f6-eaf4b48a2b61.png)

I believe most Markdown renderers would ignore single line breaks, so wrapping the text around some length shouldn't affect the rendered outcome. However, I assume Confluence handles the text within the `<p>` tag in a particular way treating those "soft" line breaks as break lines. 

Do you have any control over the HTML generation in the `mark` code to be able to handle it differently to not let Jira make those line breaks?

Many thanks.



Pull request:
Adding linebreak stripping
Adds `--strip-newlines` which collapses single newlines into a space - effectively what would happen during normal HTML rendering.  This is to address Confluence's behavior, which is to insert a `<br>`.

Closes #92


Patch:
diff --git a/README.md b/README.md
index c67235c..e73e80a 100644
--- a/README.md
+++ b/README.md
@@ -750,6 +750,7 @@ GLOBAL OPTIONS:
    --dry-run                                     resolve page and ancestry, show resulting HTML and exit. (default: false) [$MARK_DRY_RUN]
    --edit-lock, -k                               lock page editing to current user only to prevent accidental manual edits over Confluence Web UI. (default: false) [$MARK_EDIT_LOCK]
    --drop-h1, --h1_drop                          don't include the first H1 heading in Confluence output. (default: false) [$MARK_H1_DROP]
+   --strip-linebreak                             remove linebreaks inside of tags, to accomodate Confluence non-standard behavior (default: false)
    --title-from-h1, --h1_title                   extract page title from a leading H1 heading. If no H1 heading on a page exists, then title must be set in the page metadata. (default: false) [$MARK_H1_TITLE]
    --minor-edit                                  don't send notifications while updating Confluence page. (default: false) [$MARK_MINOR_EDIT]
    --color value                                 display logs in color. Possible values: auto, never. (default: "auto") [$MARK_COLOR]
@@ -759,7 +760,7 @@ GLOBAL OPTIONS:
    --password value, -p value                    use specified token for updating Confluence page. Specify - as password to read password from stdin, or your Personal access token. Username is not mandatory if personal access token is provided. For more info please see: https://developer.atlassian.com/server/confluence/confluence-server-rest-api/#authentication. [$MARK_PASSWORD]
    --target-url value, -l value                  edit specified Confluence page. If -l is not specified, file should contain metadata (see above). [$MARK_TARGET_URL]
    --base-url value, -b value, --base_url value  base URL for Confluence. Alternative option for base_url config field. [$MARK_BASE_URL]
-   --config value, -c value                      use the specified configuration file. (default: "~/.config/mark") [$MARK_CONFIG]
+   --config value, -c value                      use the specified configuration file. (default: System specific) [$MARK_CONFIG]
    --ci                                          run on CI mode. It won't fail if files are not found. (default: false) [$MARK_CI]
    --space value                                 use specified space key. If the space key is not specified, it must be set in the page metadata. [$MARK_SPACE]
    --parents value                               A list containing the parents of the document separated by parents-delimiter (default: '/'). These will be preprended to the ones defined in the document itself. [$MARK_PARENTS]
@@ -771,7 +772,7 @@ GLOBAL OPTIONS:
 ```
 
 You can store user credentials in the configuration file, which should be
-located in ~/.config/mark (or specified via `-c --config <path>`) with the following format (TOML):
+located in a system specific directory (or specified via `-c --config <path>`) with the following format (TOML):
 
 ```toml
 username = "your-email"
@@ -784,6 +785,11 @@ drop-h1 = true
 
 **NOTE**: Labels aren't supported when using `minor-edit`!
 
+**NOTE**: The system specific locations are described in here:
+https://pkg.go.dev/os#UserConfigDir.
+Currently these are:
+On Unix systems, it returns $XDG_CONFIG_HOME as specified by https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html if non-empty, else $HOME/.config. On Darwin, it returns $HOME/Library/Application Support. On Windows, it returns %AppData%. On Plan 9, it returns $home/lib.
+
 # Tricks
 
 ## Continuous Integration
diff --git a/docker-compose.yaml b/docker-compose.yaml
index 7928355..0689b9b 100644
--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -23,7 +23,7 @@ services:
       # Linux 32-bit
       # - GOOS=linux
       # - GOARCH=386
-      
+
       # Windows 64-bit
       # - GOOS=windows
       # - GOARCH=amd64
diff --git a/main.go b/main.go
index e65fce4..8f8f7f1 100644
--- a/main.go
+++ b/main.go
@@ -63,6 +63,13 @@ var flags = []cli.Flag{
 		Usage:   "don't include the first H1 heading in Confluence output.",
 		EnvVars: []string{"MARK_H1_DROP"},
 	}),
+	altsrc.NewBoolFlag(&cli.BoolFlag{
+		Name:    "strip-linebreaks",
+		Value:   false,
+		Aliases: []string{"L"},
+		Usage:   "remove linebreaks inside of tags, to accomodate non-standard Confluence behavior",
+		EnvVars: []string{"MARK_STRIP_LINEBREAK"},
+	}),
 	altsrc.NewBoolFlag(&cli.BoolFlag{
 		Name:    "title-from-h1",
 		Value:   false,
@@ -125,7 +132,7 @@ var flags = []cli.Flag{
 	&cli.StringFlag{
 		Name:      "config",
 		Aliases:   []string{"c"},
-		Value:     filepath.Join(os.Getenv("HOME"), ".config/mark"),
+		Value:     configFilePath(),
 		Usage:     "use the specified configuration file.",
 		TakesFile: true,
 		EnvVars:   []string{"MARK_CONFIG"},
@@ -182,11 +189,11 @@ func main() {
 					return altsrc.NewTomlSourceFromFile(filePath)
 				} else {
 					// Fall back to default if config is unset and path exists
-					_, err := os.Stat(filepath.Join(os.Getenv("HOME"), ".config/mark"))
+					_, err := os.Stat(configFilePath())
 					if os.IsNotExist(err) {
 						return &altsrc.MapInputSource{}, nil
 					}
-					return altsrc.NewTomlSourceFromFile(filepath.Join(os.Getenv("HOME"), ".config/mark"))
+					return altsrc.NewTomlSourceFromFile(configFilePath())
 				}
 			}),
 		EnableBashCompletion: true,
@@ -383,7 +390,7 @@ func processFile(
 			)
 		}
 
-		html, _ := mark.CompileMarkdown(markdown, stdlib, file, cCtx.String("mermaid-provider"), cCtx.Float64("mermaid-scale"), cCtx.Bool("drop-h1"))
+		html, _ := mark.CompileMarkdown(markdown, stdlib, file, cCtx.String("mermaid-provider"), cCtx.Float64("mermaid-scale"), cCtx.Bool("drop-h1"), cCtx.Bool("strip-linebreaks"))
 		fmt.Println(html)
 		os.Exit(0)
 	}
@@ -459,7 +466,7 @@ func processFile(
 		)
 	}
 
-	html, inlineAttachments := mark.CompileMarkdown(markdown, stdlib, file, cCtx.String("mermaid-provider"), cCtx.Float64("mermaid-scale"), cCtx.Bool("drop-h1"))
+	html, inlineAttachments := mark.CompileMarkdown(markdown, stdlib, file, cCtx.String("mermaid-provider"), cCtx.Float64("mermaid-scale"), cCtx.Bool("drop-h1"), cCtx.Bool("strip-linebreaks"))
 
 	// Resolve attachements detected from markdown
 	_, err = attachment.ResolveAttachments(
@@ -515,3 +522,11 @@ func processFile(
 
 	return target
 }
+
+func configFilePath() string {
+	fp, err := os.UserConfigDir()
+	if err != nil {
+		log.Fatal(err)
+	}
+	return filepath.Join(fp, "mark")
+}
diff --git a/pkg/mark/markdown.go b/pkg/mark/markdown.go
index f970257..996100d 100644
--- a/pkg/mark/markdown.go
+++ b/pkg/mark/markdown.go
@@ -26,11 +26,12 @@ type ConfluenceExtension struct {
 	MermaidProvider string
 	MermaidScale    float64
 	DropFirstH1     bool
+	StripNewlines   bool
 	Attachments     []attachment.Attachment
 }
 
 // NewConfluenceRenderer creates a new instance of the ConfluenceRenderer
-func NewConfluenceExtension(stdlib *stdlib.Lib, path string, mermaidProvider string, mermaidScale float64, dropFirstH1 bool) *ConfluenceExtension {
+func NewConfluenceExtension(stdlib *stdlib.Lib, path string, mermaidProvider string, mermaidScale float64, dropFirstH1 bool, stripNewlines bool) *ConfluenceExtension {
 	return &ConfluenceExtension{
 		Config:          html.NewConfig(),
 		Stdlib:          stdlib,
@@ -38,6 +39,7 @@ func NewConfluenceExtension(stdlib *stdlib.Lib, path string, mermaidProvider str
 		MermaidProvider: mermaidProvider,
 		MermaidScale:    mermaidScale,
 		DropFirstH1:     dropFirstH1,
+		StripNewlines:   stripNewlines,
 		Attachments:     []attachment.Attachment{},
 	}
 }
@@ -45,6 +47,7 @@ func NewConfluenceExtension(stdlib *stdlib.Lib, path string, mermaidProvider str
 func (c *ConfluenceExtension) Extend(m goldmark.Markdown) {
 
 	m.Renderer().AddOptions(renderer.WithNodeRenderers(
+		util.Prioritized(crenderer.NewConfluenceTextRenderer(c.StripNewlines), 100),
 		util.Prioritized(crenderer.NewConfluenceBlockQuoteRenderer(), 100),
 		util.Prioritized(crenderer.NewConfluenceCodeBlockRenderer(c.Stdlib, c.Path), 100),
 		util.Prioritized(crenderer.NewConfluenceFencedCodeBlockRenderer(c.Stdlib, &c.Attachments, c.MermaidProvider, c.MermaidScale), 100),
@@ -61,10 +64,10 @@ func (c *ConfluenceExtension) Extend(m goldmark.Markdown) {
 	))
 }
 
-func CompileMarkdown(markdown []byte, stdlib *stdlib.Lib, path string, mermaidProvider string, mermaidScale float64, dropFirstH1 bool) (string, []attachment.Attachment) {
+func CompileMarkdown(markdown []byte, stdlib *stdlib.Lib, path string, mermaidProvider string, mermaidScale float64, dropFirstH1 bool, stripNewlines bool) (string, []attachment.Attachment) {
 	log.Tracef(nil, "rendering markdown:\n%s", string(markdown))
 
-	confluenceExtension := NewConfluenceExtension(stdlib, path, mermaidProvider, mermaidScale, dropFirstH1)
+	confluenceExtension := NewConfluenceExtension(stdlib, path, mermaidProvider, mermaidScale, dropFirstH1, stripNewlines)
 
 	converter := goldmark.New(
 		goldmark.WithExtensions(
diff --git a/pkg/mark/renderer/text.go b/pkg/mark/renderer/text.go
new file mode 100644
index 0000000..56f752e
--- /dev/null
+++ b/pkg/mark/renderer/text.go
@@ -0,0 +1,80 @@
+package renderer
+
+import (
+	"unicode/utf8"
+
+	"github.com/yuin/goldmark/ast"
+	"github.com/yuin/goldmark/renderer"
+	"github.com/yuin/goldmark/renderer/html"
+	"github.com/yuin/goldmark/util"
+)
+
+// ConfluenceTextRenderer slightly alters the default goldmark behavior for
+// inline text block. It allows for soft breaks
+// (c.f. https://spec.commonmark.org/0.30/#softbreak)
+// to be rendered into HTML as either '\n' (the goldmark default)
+// or as ' '.
+// This latter option is useful for Confluence,
+// which inserts <br> tags into uploaded HTML where it sees '\n'.
+// See also https://sembr.org/ for partial motivation.
+type ConfluenceTextRenderer struct {
+	html.Config
+	softBreak rune
+}
+
+// NewConfluenceTextRenderer creates a new instance of the ConfluenceTextRenderer
+func NewConfluenceTextRenderer(stripNL bool, opts ...html.Option) renderer.NodeRenderer {
+	sb := '\n'
+	if stripNL {
+		sb = ' '
+	}
+	return &ConfluenceTextRenderer{
+		Config:    html.NewConfig(),
+		softBreak: sb,
+	}
+}
+
+// RegisterFuncs implements NodeRenderer.RegisterFuncs .
+func (r *ConfluenceTextRenderer) RegisterFuncs(reg renderer.NodeRendererFuncRegisterer) {
+	reg.Register(ast.KindText, r.renderText)
+}
+
+// This is taken from https://github.com/yuin/goldmark/blob/v1.5.6/renderer/html/html.go#L648
+// with the hardcoded '\n' for soft breaks swapped for the configurable r.softBreak
+func (r *ConfluenceTextRenderer) renderText(w util.BufWriter, source []byte, node ast.Node, entering bool) (ast.WalkStatus, error) {
+	if !entering {
+		return ast.WalkContinue, nil
+	}
+	n := node.(*ast.Text)
+	segment := n.Segment
+	if n.IsRaw() {
+		r.Writer.RawWrite(w, segment.Value(source))
+	} else {
+		value := segment.Value(source)
+		r.Writer.Write(w, value)
+		if n.HardLineBreak() || (n.SoftLineBreak() && r.HardWraps) {
+			if r.XHTML {
+				_, _ = w.WriteString("<br />\n")
+			} else {
+				_, _ = w.WriteString("<br>\n")
+			}
+		} else if n.SoftLineBreak() {
+			if r.EastAsianLineBreaks && len(value) != 0 {
+				sibling := node.NextSibling()
+				if sibling != nil && sibling.Kind() == ast.KindText {
+					if siblingText := sibling.(*ast.Text).Text(source); len(siblingText) != 0 {
+						thisLastRune := util.ToRune(value, len(value)-1)
+						siblingFirstRune, _ := utf8.DecodeRune(siblingText)
+						if !(util.IsEastAsianWideRune(thisLastRune) &&
+							util.IsEastAsianWideRune(siblingFirstRune)) {
+							_ = w.WriteByte(byte(r.softBreak))
+						}
+					}
+				}
+			} else {
+				_ = w.WriteByte(byte(r.softBreak))
+			}
+		}
+	}
+	return ast.WalkContinue, nil
+}


Test patch:
diff --git a/pkg/mark/markdown_test.go b/pkg/mark/markdown_test.go
index 79edc82..eaf18cd 100644
--- a/pkg/mark/markdown_test.go
+++ b/pkg/mark/markdown_test.go
@@ -10,6 +10,24 @@ import (
 	"github.com/stretchr/testify/assert"
 )
 
+func loadData(t *testing.T, filename, variant string) ([]byte, string, []byte) {
+	t.Helper()
+	basename := filepath.Base(filename)
+	testname := strings.TrimSuffix(basename, ".md")
+	htmlname := filepath.Join(filepath.Dir(filename), testname+variant+".html")
+
+	markdown, err := os.ReadFile(filename)
+	if err != nil {
+		panic(err)
+	}
+	html, err := os.ReadFile(htmlname)
+	if err != nil {
+		panic(err)
+	}
+
+	return markdown, htmlname, html
+}
+
 func TestCompileMarkdown(t *testing.T) {
 	test := assert.New(t)
 
@@ -19,24 +37,50 @@ func TestCompileMarkdown(t *testing.T) {
 	}
 
 	for _, filename := range testcases {
-		basename := filepath.Base(filename)
-		testname := strings.TrimSuffix(basename, ".md")
-		htmlname := filepath.Join(filepath.Dir(filename), testname+".html")
-
-		markdown, err := os.ReadFile(filename)
+		lib, err := stdlib.New(nil)
 		if err != nil {
 			panic(err)
 		}
-		html, err := os.ReadFile(htmlname)
+		markdown, htmlname, html := loadData(t, filename, "")
+		actual, _ := CompileMarkdown(markdown, lib, filename, "", 1.0, false, false)
+		test.EqualValues(string(html), actual, filename+" vs "+htmlname)
+	}
+}
+
+func TestCompileMarkdownDropH1(t *testing.T) {
+	test := assert.New(t)
+
+	testcases, err := filepath.Glob("testdata/*.md")
+	if err != nil {
+		panic(err)
+	}
+
+	for _, filename := range testcases {
+		lib, err := stdlib.New(nil)
 		if err != nil {
 			panic(err)
 		}
+		markdown, htmlname, html := loadData(t, filename, "-droph1")
+		actual, _ := CompileMarkdown(markdown, lib, filename, "", 1.0, true, false)
+		test.EqualValues(string(html), actual, filename+" vs "+htmlname)
+	}
+}
+
+func TestCompileMarkdownStripNewlines(t *testing.T) {
+	test := assert.New(t)
 
+	testcases, err := filepath.Glob("testdata/*.md")
+	if err != nil {
+		panic(err)
+	}
+
+	for _, filename := range testcases {
 		lib, err := stdlib.New(nil)
 		if err != nil {
 			panic(err)
 		}
-		actual, _ := CompileMarkdown(markdown, lib, filename, "", 1.0, false)
+		markdown, htmlname, html := loadData(t, filename, "-stripnewlines")
+		actual, _ := CompileMarkdown(markdown, lib, filename, "", 1.0, false, true)
 		test.EqualValues(string(html), actual, filename+" vs "+htmlname)
 	}
 }
diff --git a/pkg/mark/testdata/codes-droph1.html b/pkg/mark/testdata/codes-droph1.html
new file mode 100644
index 0000000..733410b
--- /dev/null
+++ b/pkg/mark/testdata/codes-droph1.html
@@ -0,0 +1,20 @@
+<p><code>inline</code></p>
+<ac:structured-macro ac:name="code"><ac:parameter ac:name="language"></ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[some code]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">bash</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[code bash]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">bash</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[with a newline
+]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">unknown</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[unknown code]]></ac:plain-text-body></ac:structured-macro><p>text
+text 2</p>
+<ac:structured-macro ac:name="code"><ac:parameter ac:name="language">unknown</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[unknown code 2]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">sh</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:parameter ac:name="title">A b c</ac:parameter><ac:plain-text-body><![CDATA[no-collapse-title]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">bash</ac:parameter><ac:parameter ac:name="collapse">true</ac:parameter><ac:parameter ac:name="title">A b c</ac:parameter><ac:plain-text-body><![CDATA[collapse-and-title]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">c</ac:parameter><ac:parameter ac:name="collapse">true</ac:parameter><ac:plain-text-body><![CDATA[collapse-no-title]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">nested</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[code
+``` more code ```
+even more code]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language"></ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[indented code block
+with multiple lines]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="cloudscript-confluence-mermaid"><ac:parameter ac:name="showSource">true</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[graph TD;
+    A-->B;
+    A-->C;
+    B-->D;
+    C-->D;]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="cloudscript-confluence-mermaid"><ac:parameter ac:name="showSource">true</ac:parameter><ac:parameter ac:name="collapse">true</ac:parameter><ac:parameter ac:name="title">my mermaid graph</ac:parameter><ac:plain-text-body><![CDATA[graph TD;
+    A-->B;
+    A-->C;
+    B-->D;
+    C-->D;]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="cloudscript-confluence-mermaid"><ac:parameter ac:name="showSource">true</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:parameter ac:name="title">my mermaid graph</ac:parameter><ac:plain-text-body><![CDATA[graph TD;
+    A-->B;
+    A-->C;
+    B-->D;
+    C-->D;]]></ac:plain-text-body></ac:structured-macro>
\ No newline at end of file
diff --git a/pkg/mark/testdata/codes-stripnewlines.html b/pkg/mark/testdata/codes-stripnewlines.html
new file mode 100644
index 0000000..63eff28
--- /dev/null
+++ b/pkg/mark/testdata/codes-stripnewlines.html
@@ -0,0 +1,19 @@
+<p><code>inline</code></p>
+<ac:structured-macro ac:name="code"><ac:parameter ac:name="language"></ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[some code]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">bash</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[code bash]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">bash</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[with a newline
+]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">unknown</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[unknown code]]></ac:plain-text-body></ac:structured-macro><p>text text 2</p>
+<ac:structured-macro ac:name="code"><ac:parameter ac:name="language">unknown</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[unknown code 2]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">sh</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:parameter ac:name="title">A b c</ac:parameter><ac:plain-text-body><![CDATA[no-collapse-title]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">bash</ac:parameter><ac:parameter ac:name="collapse">true</ac:parameter><ac:parameter ac:name="title">A b c</ac:parameter><ac:plain-text-body><![CDATA[collapse-and-title]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">c</ac:parameter><ac:parameter ac:name="collapse">true</ac:parameter><ac:plain-text-body><![CDATA[collapse-no-title]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language">nested</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[code
+``` more code ```
+even more code]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="code"><ac:parameter ac:name="language"></ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[indented code block
+with multiple lines]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="cloudscript-confluence-mermaid"><ac:parameter ac:name="showSource">true</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:plain-text-body><![CDATA[graph TD;
+    A-->B;
+    A-->C;
+    B-->D;
+    C-->D;]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="cloudscript-confluence-mermaid"><ac:parameter ac:name="showSource">true</ac:parameter><ac:parameter ac:name="collapse">true</ac:parameter><ac:parameter ac:name="title">my mermaid graph</ac:parameter><ac:plain-text-body><![CDATA[graph TD;
+    A-->B;
+    A-->C;
+    B-->D;
+    C-->D;]]></ac:plain-text-body></ac:structured-macro><ac:structured-macro ac:name="cloudscript-confluence-mermaid"><ac:parameter ac:name="showSource">true</ac:parameter><ac:parameter ac:name="collapse">false</ac:parameter><ac:parameter ac:name="title">my mermaid graph</ac:parameter><ac:plain-text-body><![CDATA[graph TD;
+    A-->B;
+    A-->C;
+    B-->D;
+    C-->D;]]></ac:plain-text-body></ac:structured-macro>
\ No newline at end of file
diff --git a/pkg/mark/testdata/header-droph1.html b/pkg/mark/testdata/header-droph1.html
new file mode 100644
index 0000000..c63a140
--- /dev/null
+++ b/pkg/mark/testdata/header-droph1.html
@@ -0,0 +1,6 @@
+<h2 id="b">b</h2>
+<h3 id="c">c</h3>
+<h4 id="d">d</h4>
+<h5 id="e">e</h5>
+<h1 id="f">f</h1>
+<h2 id="g">g</h2>
diff --git a/pkg/mark/testdata/header-stripnewlines.html b/pkg/mark/testdata/header-stripnewlines.html
new file mode 100644
index 0000000..566c4ae
--- /dev/null
+++ b/pkg/mark/testdata/header-stripnewlines.html
@@ -0,0 +1,7 @@
+<h1 id="a">a</h1>
+<h2 id="b">b</h2>
+<h3 id="c">c</h3>
+<h4 id="d">d</h4>
+<h5 id="e">e</h5>
+<h1 id="f">f</h1>
+<h2 id="g">g</h2>
diff --git a/pkg/mark/testdata/issue-64-broken-link-droph1.html b/pkg/mark/testdata/issue-64-broken-link-droph1.html
new file mode 100644
index 0000000..0d39ce8
--- /dev/null
+++ b/pkg/mark/testdata/issue-64-broken-link-droph1.html
@@ -0,0 +1,1 @@
+<p><a href="Page2#Page2-Releasev71-22-Feb-2018(intern)">v71</a></p>
diff --git a/pkg/mark/testdata/issue-64-broken-link-stripnewlines.html b/pkg/mark/testdata/issue-64-broken-link-stripnewlines.html
new file mode 100644
index 0000000..0d39ce8
--- /dev/null
+++ b/pkg/mark/testdata/issue-64-broken-link-stripnewlines.html
@@ -0,0 +1,1 @@
+<p><a href="Page2#Page2-Releasev71-22-Feb-2018(intern)">v71</a></p>
diff --git a/pkg/mark/testdata/links-droph1.html b/pkg/mark/testdata/links-droph1.html
new file mode 100644
index 0000000..82fd040
--- /dev/null
+++ b/pkg/mark/testdata/links-droph1.html
@@ -0,0 +1,17 @@
+<p>Use <a href="https://example.com">https://example.com</a></p>
+<p>Use <ac:rich-text-body>aaa</ac:rich-text-body></p>
+<p>Use <ac:link><ri:page ri:content-title="Page"/><ac:plain-text-link-body><![CDATA[page link]]></ac:plain-text-link-body></ac:link></p>
+<p>Use <ac:link><ri:page ri:content-title="AnotherPage"/><ac:plain-text-link-body><![CDATA[AnotherPage]]></ac:plain-text-link-body></ac:link></p>
+<p>Use <ac:link><ri:page ri:content-title="Another Page"/><ac:plain-text-link-body><![CDATA[Another Page]]></ac:plain-text-link-body></ac:link></p>
+<p>Use <ac:link><ri:page ri:content-title="Page With Space"/><ac:plain-text-link-body><![CDATA[page link with spaces]]></ac:plain-text-link-body></ac:link></p>
+<p><ac:image ac:alt="My Image"><ri:attachment ri:filename="test.png"/></ac:image></p>
+<p><ac:image ac:alt="My External Image"><ri:url ri:value="http://confluence.atlassian.com/images/logo/confluence_48_trans.png?key1=value1&amp;key2=value2"/></ac:image></p>
+<p>Use footnotes link <sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p>
+<div class="footnotes" role="doc-endnotes">
+<hr />
+<ol>
+<li id="fn:1">
+<p>a footnote link&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
+</li>
+</ol>
+</div>
diff --git a/pkg/mark/testdata/links-stripnewlines.html b/pkg/mark/testdata/links-stripnewlines.html
new file mode 100644
index 0000000..82fd040
--- /dev/null
+++ b/pkg/mark/testdata/links-stripnewlines.html
@@ -0,0 +1,17 @@
+<p>Use <a href="https://example.com">https://example.com</a></p>
+<p>Use <ac:rich-text-body>aaa</ac:rich-text-body></p>
+<p>Use <ac:link><ri:page ri:content-title="Page"/><ac:plain-text-link-body><![CDATA[page link]]></ac:plain-text-link-body></ac:link></p>
+<p>Use <ac:link><ri:page ri:content-title="AnotherPage"/><ac:plain-text-link-body><![CDATA[AnotherPage]]></ac:plain-text-link-body></ac:link></p>
+<p>Use <ac:link><ri:page ri:content-title="Another Page"/><ac:plain-text-link-body><![CDATA[Another Page]]></ac:plain-text-link-body></ac:link></p>
+<p>Use <ac:link><ri:page ri:content-title="Page With Space"/><ac:plain-text-link-body><![CDATA[page link with spaces]]></ac:plain-text-link-body></ac:link></p>
+<p><ac:image ac:alt="My Image"><ri:attachment ri:filename="test.png"/></ac:image></p>
+<p><ac:image ac:alt="My External Image"><ri:url ri:value="http://confluence.atlassian.com/images/logo/confluence_48_trans.png?key1=value1&amp;key2=value2"/></ac:image></p>
+<p>Use footnotes link <sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p>
+<div class="footnotes" role="doc-endnotes">
+<hr />
+<ol>
+<li id="fn:1">
+<p>a footnote link&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p>
+</li>
+</ol>
+</div>
diff --git a/pkg/mark/testdata/lists-droph1.html b/pkg/mark/testdata/lists-droph1.html
new file mode 100644
index 0000000..af27c54
--- /dev/null
+++ b/pkg/mark/testdata/lists-droph1.html
@@ -0,0 +1,21 @@
+<ul>
+<li>dash 1-1</li>
+<li>dash 1-2</li>
+<li>dash 1-3
+<ul>
+<li>dash 1-3-1</li>
+<li>dash 1-3-2</li>
+<li>dash 1-3-3
+<ul>
+<li>dash 1-3-3-1</li>
+</ul>
+</li>
+</ul>
+</li>
+</ul>
+<p>text</p>
+<ul>
+<li>a</li>
+<li>b</li>
+<li>c</li>
+</ul>
diff --git a/pkg/mark/testdata/lists-stripnewlines.html b/pkg/mark/testdata/lists-stripnewlines.html
new file mode 100644
index 0000000..af27c54
--- /dev/null
+++ b/pkg/mark/testdata/lists-stripnewlines.html
@@ -0,0 +1,21 @@
+<ul>
+<li>dash 1-1</li>
+<li>dash 1-2</li>
+<li>dash 1-3
+<ul>
+<li>dash 1-3-1</li>
+<li>dash 1-3-2</li>
+<li>dash 1-3-3
+<ul>
+<li>dash 1-3-3-1</li>
+</ul>
+</li>
+</ul>
+</li>
+</ul>
+<p>text</p>
+<ul>
+<li>a</li>
+<li>b</li>
+<li>c</li>
+</ul>
diff --git a/pkg/mark/testdata/macro-include-droph1.html b/pkg/mark/testdata/macro-include-droph1.html
new file mode 100644
index 0000000..8d8ba05
--- /dev/null
+++ b/pkg/mark/testdata/macro-include-droph1.html
@@ -0,0 +1,6 @@
+<p><foo>bar</foo></p>
+<p><ac:structured-macro ac:name="info">
+<ac:parameter ac:name="icon">true</ac:parameter>
+<ac:parameter ac:name="title">Attention</ac:parameter>
+<ac:rich-text-body>This is an info!</ac:rich-text-body>
+</ac:structured-macro></p>
diff --git a/pkg/mark/testdata/macro-include-stripnewlines.html b/pkg/mark/testdata/macro-include-stripnewlines.html
new file mode 100644
index 0000000..52e4af7
--- /dev/null
+++ b/pkg/mark/testdata/macro-include-stripnewlines.html
@@ -0,0 +1,2 @@
+<p><foo>bar</foo></p>
+<p><ac:structured-macro ac:name="info"> <ac:parameter ac:name="icon">true</ac:parameter> <ac:parameter ac:name="title">Attention</ac:parameter> <ac:rich-text-body>This is an info!</ac:rich-text-body> </ac:structured-macro></p>
diff --git a/pkg/mark/testdata/newlines-droph1.html b/pkg/mark/testdata/newlines-droph1.html
new file mode 100644
index 0000000..b97e510
--- /dev/null
+++ b/pkg/mark/testdata/newlines-droph1.html
@@ -0,0 +1,10 @@
+<p>one-1
+one-2</p>
+<p>two-1</p>
+<p>two-2</p>
+<p>three-1</p>
+<p>three-2</p>
+<p>space-1
+space-2</p>
+<p>2space-1<br />
+2space-2</p>
diff --git a/pkg/mark/testdata/newlines-stripnewlines.html b/pkg/mark/testdata/newlines-stripnewlines.html
new file mode 100644
index 0000000..2fd4dc0
--- /dev/null
+++ b/pkg/mark/testdata/newlines-stripnewlines.html
@@ -0,0 +1,8 @@
+<p>one-1 one-2</p>
+<p>two-1</p>
+<p>two-2</p>
+<p>three-1</p>
+<p>three-2</p>
+<p>space-1 space-2</p>
+<p>2space-1<br />
+2space-2</p>
diff --git a/pkg/mark/testdata/pagelayout-droph1.html b/pkg/mark/testdata/pagelayout-droph1.html
new file mode 100644
index 0000000..66f5783
--- /dev/null
+++ b/pkg/mark/testdata/pagelayout-droph1.html
@@ -0,0 +1,18 @@
+<ac:layout>
+<ac:layout-section type="three_with_sidebars">
+<ac:layout-cell>
+<p>More Content</p>
+</ac:layout-cell>
+<ac:layout-cell>
+<p>More Content</p>
+</ac:layout-cell>
+<ac:layout-cell>
+<p>Even More Content</p>
+</ac:layout-cell>
+</ac:layout-section>
+<ac:layout-section type="single">
+<ac:layout-cell>
+<p>Still More Content</p>
+</ac:layout-cell>
+</ac:layout-section>
+</ac:layout>
diff --git a/pkg/mark/testdata/pagelayout-stripnewlines.html b/pkg/mark/testdata/pagelayout-stripnewlines.html
new file mode 100644
index 0000000..66f5783
--- /dev/null
+++ b/pkg/mark/testdata/pagelayout-stripnewlines.html
@@ -0,0 +1,18 @@
+<ac:layout>
+<ac:layout-section type="three_with_sidebars">
+<ac:layout-cell>
+<p>More Content</p>
+</ac:layout-cell>
+<ac:layout-cell>
+<p>More Content</p>
+</ac:layout-cell>
+<ac:layout-cell>
+<p>Even More Content</p>
+</ac:layout-cell>
+</ac:layout-section>
+<ac:layout-section type="single">
+<ac:layout-cell>
+<p>Still More Content</p>
+</ac:layout-cell>
+</ac:layout-section>
+</ac:layout>
diff --git a/pkg/mark/testdata/quotes-droph1.html b/pkg/mark/testdata/quotes-droph1.html
new file mode 100644
index 0000000..d8fdcdf
--- /dev/null
+++ b/pkg/mark/testdata/quotes-droph1.html
@@ -0,0 +1,34 @@
+<h2 id="first-heading">First Heading</h2>
+<ac:structured-macro ac:name="note"><ac:parameter ac:name="icon">true</ac:parameter><ac:rich-text-body>
+<p><strong>NOTES:</strong></p>
+<ol>
+<li>Note number one</li>
+<li>Note number two</li>
+</ol>
+<blockquote>
+<p>a
+b</p>
+</blockquote>
+<p><strong>Warn (Should not be picked as blockquote type)</strong></p>
+</ac:rich-text-body></ac:structured-macro>
+<h2 id="second-heading">Second Heading</h2>
+<ac:structured-macro ac:name="warn"><ac:parameter ac:name="icon">true</ac:parameter><ac:rich-text-body>
+<p><strong>Warn</strong></p>
+<ul>
+<li>Warn bullet 1</li>
+<li>Warn bullet 2</li>
+</ul>
+</ac:rich-text-body></ac:structured-macro>
+<ul>
+<li>Regular list
+that runs long</li>
+</ul>
+<h2 id="third-heading">Third Heading</h2>
+<ac:structured-macro ac:name="info"><ac:parameter ac:name="icon">true</ac:parameter><ac:rich-text-body>
+<!-- Info -->
+<p>Test</p>
+</ac:rich-text-body></ac:structured-macro>
+<h2 id="simple-blockquote">Simple Blockquote</h2>
+<blockquote>
+<p>This paragraph is a simple blockquote</p>
+</blockquote>
diff --git a/pkg/mark/testdata/quotes-stripnewlines.html b/pkg/mark/testdata/quotes-stripnewlines.html
new file mode 100644
index 0000000..7a43019
--- /dev/null
+++ b/pkg/mark/testdata/quotes-stripnewlines.html
@@ -0,0 +1,33 @@
+<h1 id="main-heading">Main Heading</h1>
+<h2 id="first-heading">First Heading</h2>
+<ac:structured-macro ac:name="note"><ac:parameter ac:name="icon">true</ac:parameter><ac:rich-text-body>
+<p><strong>NOTES:</strong></p>
+<ol>
+<li>Note number one</li>
+<li>Note number two</li>
+</ol>
+<blockquote>
+<p>a b</p>
+</blockquote>
+<p><strong>Warn (Should not be picked as blockquote type)</strong></p>
+</ac:rich-text-body></ac:structured-macro>
+<h2 id="second-heading">Second Heading</h2>
+<ac:structured-macro ac:name="warn"><ac:parameter ac:name="icon">true</ac:parameter><ac:rich-text-body>
+<p><strong>Warn</strong></p>
+<ul>
+<li>Warn bullet 1</li>
+<li>Warn bullet 2</li>
+</ul>
+</ac:rich-text-body></ac:structured-macro>
+<ul>
+<li>Regular list that runs long</li>
+</ul>
+<h2 id="third-heading">Third Heading</h2>
+<ac:structured-macro ac:name="info"><ac:parameter ac:name="icon">true</ac:parameter><ac:rich-text-body>
+<!-- Info -->
+<p>Test</p>
+</ac:rich-text-body></ac:structured-macro>
+<h2 id="simple-blockquote">Simple Blockquote</h2>
+<blockquote>
+<p>This paragraph is a simple blockquote</p>
+</blockquote>
diff --git a/pkg/mark/testdata/quotes.html b/pkg/mark/testdata/quotes.html
index acb61d6..c1519c7 100644
--- a/pkg/mark/testdata/quotes.html
+++ b/pkg/mark/testdata/quotes.html
@@ -20,6 +20,10 @@ b</p>
 <li>Warn bullet 2</li>
 </ul>
 </ac:rich-text-body></ac:structured-macro>
+<ul>
+<li>Regular list
+that runs long</li>
+</ul>
 <h2 id="third-heading">Third Heading</h2>
 <ac:structured-macro ac:name="info"><ac:parameter ac:name="icon">true</ac:parameter><ac:rich-text-body>
 <!-- Info -->
diff --git a/pkg/mark/testdata/quotes.md b/pkg/mark/testdata/quotes.md
index f6b9958..995f67f 100644
--- a/pkg/mark/testdata/quotes.md
+++ b/pkg/mark/testdata/quotes.md
@@ -19,6 +19,8 @@
 > * Warn bullet 1
 > * Warn bullet 2
 
+* Regular list
+  that runs long
 
 ## Third Heading
 > <!-- Info -->
diff --git a/pkg/mark/testdata/table-droph1.html b/pkg/mark/testdata/table-droph1.html
new file mode 100644
index 0000000..9b4f80a
--- /dev/null
+++ b/pkg/mark/testdata/table-droph1.html
@@ -0,0 +1,14 @@
+<table>
+<thead>
+<tr>
+<th>HEADER1</th>
+<th>HEADER2</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>row1</td>
+<td>row2</td>
+</tr>
+</tbody>
+</table>
diff --git a/pkg/mark/testdata/table-stripnewlines.html b/pkg/mark/testdata/table-stripnewlines.html
new file mode 100644
index 0000000..9b4f80a
--- /dev/null
+++ b/pkg/mark/testdata/table-stripnewlines.html
@@ -0,0 +1,14 @@
+<table>
+<thead>
+<tr>
+<th>HEADER1</th>
+<th>HEADER2</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>row1</td>
+<td>row2</td>
+</tr>
+</tbody>
+</table>
diff --git a/pkg/mark/testdata/tags-droph1.html b/pkg/mark/testdata/tags-droph1.html
new file mode 100644
index 0000000..a5161db
--- /dev/null
+++ b/pkg/mark/testdata/tags-droph1.html
@@ -0,0 +1,6 @@
+<p><b>bold</b>
+<strong>bold</strong></p>
+<p><i>vitalik</i>
+<em>vitalik</em></p>
+<p>
strikethrough

+<del>strikethrough</del></p>
diff --git a/pkg/mark/testdata/tags-stripnewlines.html b/pkg/mark/testdata/tags-stripnewlines.html
new file mode 100644
index 0000000..cc0698b
--- /dev/null
+++ b/pkg/mark/testdata/tags-stripnewlines.html
@@ -0,0 +1,3 @@
+<p><b>bold</b> <strong>bold</strong></p>
+<p><i>vitalik</i> <em>vitalik</em></p>
+<p>
strikethrough
 <del>strikethrough</del></p>