Repository: goccy/go-yaml
Pull request #499 (2024-11-02 10:26:53)

Fix parsing of invalid document
ref https://github.com/goccy/go-yaml/issues/191

codecov-commenter: :warning: Please install the !['codecov app svg image'](https://github.com/codecov/engineering-team/assets/152432831/e90313f4-9d3a-4b63-8b54-cfe14e7ec20d) to ensure uploads and comments are reliably processed by Codecov.

## [Codecov](https://app.codecov.io/gh/goccy/go-yaml/pull/499?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Masaaki+Goshima) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 76.56%. Comparing base [(`61505f3`)](https://app.codecov.io/gh/goccy/go-yaml/commit/61505f3849e3b6c9d04340ac1842301eb0cec726?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Masaaki+Goshima) to head [(`873d14e`)](https://app.codecov.io/gh/goccy/go-yaml/commit/873d14ea80353c3970b26b33229f987109ef7602?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Masaaki+Goshima).

:exclamation: Your organization needs to install the [Codecov GitHub app](https://github.com/apps/codecov/installations/select_target) to enable full functionality.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##           master     #499   +/-   ##
=======================================
  Coverage   76.55%   76.56%           
=======================================
  Files          18       18           
  Lines        6027     6037   +10     
=======================================
+ Hits         4614     4622    +8     
- Misses       1106     1108    +2     
  Partials      307      307           
```

</details>

Patch:
diff --git a/scanner/scanner.go b/scanner/scanner.go
index 4e7d0aa..4426b14 100644
--- a/scanner/scanner.go
+++ b/scanner/scanner.go
@@ -837,6 +837,9 @@ func (s *Scanner) scanLiteralHeaderOption(ctx *Context) error {
 			case "", "+", "-",
 				"0", "1", "2", "3", "4", "5", "6", "7", "8", "9":
 				hasComment := len(opt) < orgOptLen
+				if s.column == 1 {
+					s.lastDelimColumn = 1
+				}
 				if header == '|' {
 					if hasComment {
 						commentLen := orgOptLen - len(opt)
@@ -961,6 +964,13 @@ func (s *Scanner) scan(ctx *Context) error {
 				if tk := ctx.lastToken(); tk != nil {
 					// If literal/folded content is empty, no string token is added.
 					// Therefore, add an empty string token.
+					// But if literal/folded token column is 1, it is invalid at down state.
+					if tk.Position.Column == 1 {
+						return ErrInvalidToken(
+							"could not find document",
+							token.Invalid(string(ctx.obuf), s.pos()),
+						)
+					}
 					if tk.Type != token.StringType {
 						ctx.addToken(token.String("", "", s.pos()))
 					}


Test patch:
diff --git a/lexer/lexer_test.go b/lexer/lexer_test.go
index b833221..d36215d 100644
--- a/lexer/lexer_test.go
+++ b/lexer/lexer_test.go
@@ -2421,6 +2421,14 @@ a: |invalid`,
 			name: "invalid double-quoted",
 			src:  `a: "\"key\": \"value:\"`,
 		},
+		{
+			name: "invalid document folded",
+			src:  ">\n>",
+		},
+		{
+			name: "invalid document number",
+			src:  ">\n1",
+		},
 	}
 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
diff --git a/parser/parser_test.go b/parser/parser_test.go
index 3c85e49..cf77aa2 100644
--- a/parser/parser_test.go
+++ b/parser/parser_test.go
@@ -1099,6 +1099,33 @@ b: - 2
 [1:8] ',' or ']' must be specified
 >  1 | foo: [$[should not be allowed]]
               ^
+`,
+		},
+		{
+			">\n>",
+			`
+[2:1] found invalid token
+   1 | >
+>  2 | >
+       ^
+`,
+		},
+		{
+			">\n1",
+			`
+[2:1] found invalid token
+   1 | >
+>  2 | 1
+       ^
+`,
+		},
+		{
+			"|\n1",
+			`
+[2:1] found invalid token
+   1 | |
+>  2 | 1
+       ^
 `,
 		},
 	}