Repository: goccy/go-yaml
Pull request #555 (2024-11-29 13:19:29)

Replace fmt.Sprint with faster alternatives
The PR replaces `fmt.Sprint.*` statements with faster alternatives suggested by the [`perfsprint`](https://golangci-lint.run/usage/linters/#perfsprint) linter:

- Replace `fmt.Sprint(v)` with `strconv.FormatInt(v, 10)`, where `v` is `int64`.
- Replace `fmt.Sprint(v)` with `strconv.FormatUint(v, 10)`, where `v` is `uint64`.
- Replace `fmt.Sprint(v)` with `strconv.FormatBool(v)`, where `v` is `bool`.
- Replace `fmt.Errorf(msg)` with `errors.New(msg)`.

codecov-commenter: :warning: Please install the !['codecov app svg image'](https://github.com/codecov/engineering-team/assets/152432831/e90313f4-9d3a-4b63-8b54-cfe14e7ec20d) to ensure uploads and comments are reliably processed by Codecov.

## [Codecov](https://app.codecov.io/gh/goccy/go-yaml/pull/555?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Masaaki+Goshima) Report
Attention: Patch coverage is `87.50000%` with `1 line` in your changes missing coverage. Please review.
> Project coverage is 77.00%. Comparing base [(`01f4bac`)](https://app.codecov.io/gh/goccy/go-yaml/commit/01f4bac35ca0bba7bf113a436a34b4df29a433a5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Masaaki+Goshima) to head [(`c0f4ed5`)](https://app.codecov.io/gh/goccy/go-yaml/commit/c0f4ed5cdcd89663f53a35ac8175ef757b83dc9a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Masaaki+Goshima).

:exclamation: Your organization needs to install the [Codecov GitHub app](https://github.com/apps/codecov/installations/select_target) to enable full functionality.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##           master     #555   +/-   ##
=======================================
  Coverage   77.00%   77.00%           
=======================================
  Files          21       21           
  Lines        7270     7270           
=======================================
  Hits         5598     5598           
  Misses       1277     1277           
  Partials      395      395           
```

</details>

Patch:
diff --git a/.golangci.yml b/.golangci.yml
index ec8d731..9ba6690 100644
--- a/.golangci.yml
+++ b/.golangci.yml
@@ -18,6 +18,12 @@ linters-settings:
       - tests
   misspell:
     locale: US
+  perfsprint:
+    int-conversion: false
+    err-error: false
+    errorf: true
+    sprintf1: false
+    strconcat: false
   staticcheck:
     checks: ["all", "-ST1000", "-ST1005"]
 
@@ -31,6 +37,7 @@ linters:
     - govet
     - ineffassign
     - misspell
+    - perfsprint
     - staticcheck
     - typecheck
     - unused
diff --git a/README.md b/README.md
index 30ff96a..afcc802 100644
--- a/README.md
+++ b/README.md
@@ -9,30 +9,52 @@
 
 ## This library has **NO** relation to the go-yaml/yaml library
 
-This library is developed from scratch to replace [`go-yaml/yaml`](https://github.com/go-yaml/yaml).
-If you're looking for a better YAML library, this one should be helpful.
+> [!IMPORTANT]
+> This library is developed from scratch to replace [`go-yaml/yaml`](https://github.com/go-yaml/yaml).
+> If you're looking for a better YAML library, this one should be helpful.
 
 # Why a new library?
 
-As of this writing, there already exists a de facto standard library for YAML processing for Go: [https://github.com/go-yaml/yaml](https://github.com/go-yaml/yaml). However we feel that some features are lacking, namely:
+As of this writing, there already exists a de facto standard library for YAML processing for Go: [https://github.com/go-yaml/yaml](https://github.com/go-yaml/yaml). However, we believe that a new YAML library is necessary for the following reasons:
 
-- Pretty format for error notifications
-- Direct manipulation of YAML abstract syntax tree
-- Support for `Anchor` and `Alias` when marshaling
-- Allow referencing elements declared in another file via anchors
+- Not actively maintained
+- `go-yaml/yaml` has ported the libyaml written in C to Go, so the source code is not written in Go style
+- There is a lot of content that cannot be parsed
+- YAML is often used for configuration, and it is common to include validation along with it. However, the errors in `go-yaml/yaml` are not intuitive, and it is difficult to provide meaningful validation errors
+- When creating tools that use YAML, there are cases where reversible transformation of YAML is required. However, to perform reversible transformations of content that includes Comments or Anchors/Aliases, manipulating the AST is the only option
+- Non-intuitive [Marshaler](https://pkg.go.dev/gopkg.in/yaml.v3#Marshaler) / [Unmarshaler](https://pkg.go.dev/gopkg.in/yaml.v3#Unmarshaler)
+
+By the way, libraries such as [ghodss/yaml](https://github.com/ghodss/yaml) and [sigs.k8s.io/yaml](https://github.com/kubernetes-sigs/yaml) also depend on go-yaml/yaml, so if you are using these libraries, the same issues apply: they cannot parse things that go-yaml/yaml cannot parse, and they inherit many of the problems that go-yaml/yaml has.
 
 # Features
 
+- No dependencies
+- A better parser than `go-yaml/yaml`. 
+  - [Support recursive processing](https://github.com/apple/device-management/blob/release/docs/schema.yaml)
+  - Higher coverage in the [YAML Test Suite](https://github.com/yaml/yaml-test-suite?tab=readme-ov-file)
+- Ease and sustainability of maintenance
+  - The main maintainer is [@goccy](https://github.com/goccy), but we are also building a system to develop as a team with trusted developers
+  - Since it is written from scratch, the code is easy to read for Gophers
+- An API structure that allows the use of not only `Encoder`/`Decoder` but also `Tokenizer` and `Parser` functionalities.
+  - [lexer.Tokenize](https://pkg.go.dev/github.com/goccy/go-yaml@v1.15.4/lexer#Tokenize)
+  - [parser.Parse](https://pkg.go.dev/github.com/goccy/go-yaml@v1.15.4/parser#Parse)
+- Filtering, replacing, and merging YAML content using YAML Path
+- Reversible transformation without using the AST for YAML that includes Anchors, Aliases, and Comments
+- Customize the Marshal/Unmarshal behavior for primitive types and third-party library types ([RegisterCustomMarshaler](https://pkg.go.dev/github.com/goccy/go-yaml#RegisterCustomMarshaler), [RegisterCustomUnmarshaler](https://pkg.go.dev/github.com/goccy/go-yaml#RegisterCustomUnmarshaler))
+- Respects `encoding/json` behavior
+  - Accept the `json` tag. Note that not all options from the `json` tag will have significance when parsing YAML documents. If both tags exist, `yaml` tag will take precedence.
+  - [json.Marshaler](https://pkg.go.dev/encoding/json#Marshaler) style [marshaler](https://pkg.go.dev/github.com/goccy/go-yaml#BytesMarshaler)
+  - [json.Unmarshaler](https://pkg.go.dev/encoding/json#Unmarshaler) style [unmarshaler](https://pkg.go.dev/github.com/goccy/go-yaml#BytesUnmarshaler)
+  - Options for using `MarshalJSON` and `UnmarshalJSON` ([UseJSONMarshaler](https://pkg.go.dev/github.com/goccy/go-yaml#UseJSONMarshaler), [UseJSONUnmarshaler](https://pkg.go.dev/github.com/goccy/go-yaml#UseJSONUnmarshaler))
 - Pretty format for error notifications
-- Supports `Scanner` or `Lexer` or `Parser` as public API
-- Supports `Anchor` and `Alias` to Marshaler
+- Smart validation processing combined with [go-playground/validator](https://github.com/go-playground/validator)
+  - [example test code is here](https://github.com/goccy/go-yaml/blob/45889c98b0a0967240eb595a1bd6896e2f575106/testdata/validate_test.go#L12)
 - Allow referencing elements declared in another file via anchors
-- Extract value or AST by YAMLPath ( YAMLPath is like a JSONPath )
 
 # Installation
 
 ```sh
-go get -u github.com/goccy/go-yaml
+go get github.com/goccy/go-yaml
 ```
 
 # Synopsis
diff --git a/ast/ast.go b/ast/ast.go
index 52edfdf..1ad8f15 100644
--- a/ast/ast.go
+++ b/ast/ast.go
@@ -1475,6 +1475,11 @@ func (n *SequenceNode) Merge(target *SequenceNode) {
 	column := n.Start.Position.Column - target.Start.Position.Column
 	target.AddColumn(column)
 	n.Values = append(n.Values, target.Values...)
+	if len(target.ValueHeadComments) == 0 {
+		n.ValueHeadComments = append(n.ValueHeadComments, make([]*CommentGroupNode, len(target.Values))...)
+		return
+	}
+	n.ValueHeadComments = append(n.ValueHeadComments, target.ValueHeadComments...)
 }
 
 // SetIsFlowStyle set value to IsFlowStyle field recursively.
@@ -1721,8 +1726,12 @@ func (n *AliasNode) MarshalYAML() ([]byte, error) {
 // DirectiveNode type of directive node
 type DirectiveNode struct {
 	*BaseNode
+	// Start is '%' token.
 	Start *token.Token
-	Value Node
+	// Name is directive name e.g.) "YAML" or "TAG".
+	Name Node
+	// Values is directive values e.g.) "1.2" or "!!" and "tag:clarkevans.com,2002:app/".
+	Values []Node
 }
 
 // Read implements (io.Reader).Read
@@ -1740,14 +1749,21 @@ func (n *DirectiveNode) GetToken() *token.Token {
 
 // AddColumn add column number to child nodes recursively
 func (n *DirectiveNode) AddColumn(col int) {
-	if n.Value != nil {
-		n.Value.AddColumn(col)
+	if n.Name != nil {
+		n.Name.AddColumn(col)
+	}
+	for _, value := range n.Values {
+		value.AddColumn(col)
 	}
 }
 
 // String directive to text
 func (n *DirectiveNode) String() string {
-	return fmt.Sprintf("%s%s", n.Start.Value, n.Value.String())
+	values := make([]string, 0, len(n.Values))
+	for _, val := range n.Values {
+		values = append(values, val.String())
+	}
+	return strings.Join(append([]string{"%" + n.Name.String()}, values...), " ")
 }
 
 // MarshalYAML encodes to a YAML text
@@ -1758,8 +1774,9 @@ func (n *DirectiveNode) MarshalYAML() ([]byte, error) {
 // TagNode type of tag node
 type TagNode struct {
 	*BaseNode
-	Start *token.Token
-	Value Node
+	Directive *DirectiveNode
+	Start     *token.Token
+	Value     Node
 }
 
 func (n *TagNode) GetValue() any {
@@ -1935,7 +1952,10 @@ func Walk(v Visitor, node Node) {
 		Walk(v, n.Value)
 	case *DirectiveNode:
 		walkComment(v, n.BaseNode)
-		Walk(v, n.Value)
+		Walk(v, n.Name)
+		for _, value := range n.Values {
+			Walk(v, value)
+		}
 	case *TagNode:
 		walkComment(v, n.BaseNode)
 		Walk(v, n.Value)
@@ -2021,7 +2041,14 @@ func (f *parentFinder) walk(parent, node Node) Node {
 	case *LiteralNode:
 		return f.walk(node, n.Value)
 	case *DirectiveNode:
-		return f.walk(node, n.Value)
+		if found := f.walk(node, n.Name); found != nil {
+			return found
+		}
+		for _, value := range n.Values {
+			if found := f.walk(node, value); found != nil {
+				return found
+			}
+		}
 	case *TagNode:
 		return f.walk(node, n.Value)
 	case *DocumentNode:
diff --git a/decode.go b/decode.go
index f02fc05..024e9ba 100644
--- a/decode.go
+++ b/decode.go
@@ -354,6 +354,16 @@ func (d *Decoder) nodeToValue(node ast.Node) (any, error) {
 	case *ast.NanNode:
 		return n.GetValue(), nil
 	case *ast.TagNode:
+		if n.Directive != nil {
+			v, err := d.nodeToValue(n.Value)
+			if err != nil {
+				return nil, err
+			}
+			if v == nil {
+				return "", nil
+			}
+			return fmt.Sprint(v), nil
+		}
 		switch token.ReservedTagKeyword(n.Start.Value) {
 		case token.TimestampTag:
 			t, _ := d.castToTime(n.Value)
@@ -405,7 +415,14 @@ func (d *Decoder) nodeToValue(node ast.Node) (any, error) {
 			}
 			return nil, errors.ErrSyntax(fmt.Sprintf("cannot convert %q to boolean", fmt.Sprint(v)), n.Value.GetToken())
 		case token.StringTag:
-			return d.nodeToValue(n.Value)
+			v, err := d.nodeToValue(n.Value)
+			if err != nil {
+				return nil, err
+			}
+			if v == nil {
+				return "", nil
+			}
+			return fmt.Sprint(v), nil
 		case token.MappingTag:
 			return d.nodeToValue(n.Value)
 		default:
@@ -676,13 +693,13 @@ func (d *Decoder) convertValue(v reflect.Value, typ reflect.Type, src ast.Node)
 	// cast value to string
 	switch v.Type().Kind() {
 	case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
-		return reflect.ValueOf(fmt.Sprint(v.Int())), nil
+		return reflect.ValueOf(strconv.FormatInt(v.Int(), 10)), nil
 	case reflect.Float32, reflect.Float64:
 		return reflect.ValueOf(fmt.Sprint(v.Float())), nil
 	case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
-		return reflect.ValueOf(fmt.Sprint(v.Uint())), nil
+		return reflect.ValueOf(strconv.FormatUint(v.Uint(), 10)), nil
 	case reflect.Bool:
-		return reflect.ValueOf(fmt.Sprint(v.Bool())), nil
+		return reflect.ValueOf(strconv.FormatBool(v.Bool())), nil
 	}
 	if !v.Type().ConvertibleTo(typ) {
 		return reflect.Zero(typ), errors.ErrTypeMismatch(typ, v.Type(), src.GetToken())
@@ -936,7 +953,7 @@ func (d *Decoder) decodeByUnmarshaler(ctx context.Context, dst reflect.Value, sr
 		}
 	}
 
-	return fmt.Errorf("does not implemented Unmarshaler")
+	return errors.New("does not implemented Unmarshaler")
 }
 
 var (
diff --git a/encode.go b/encode.go
index 97c1fa5..85080ff 100644
--- a/encode.go
+++ b/encode.go
@@ -500,12 +500,12 @@ func (e *Encoder) encodeNil() *ast.NullNode {
 }
 
 func (e *Encoder) encodeInt(v int64) *ast.IntegerNode {
-	value := fmt.Sprint(v)
+	value := strconv.FormatInt(v, 10)
 	return ast.Integer(token.New(value, value, e.pos(e.column)))
 }
 
 func (e *Encoder) encodeUint(v uint64) *ast.IntegerNode {
-	value := fmt.Sprint(v)
+	value := strconv.FormatUint(v, 10)
 	return ast.Integer(token.New(value, value, e.pos(e.column)))
 }
 
@@ -556,7 +556,7 @@ func (e *Encoder) encodeString(v string, column int) *ast.StringNode {
 }
 
 func (e *Encoder) encodeBool(v bool) *ast.BoolNode {
-	value := fmt.Sprint(v)
+	value := strconv.FormatBool(v)
 	return ast.Bool(token.New(value, value, e.pos(e.column)))
 }
 
diff --git a/parser/parser.go b/parser/parser.go
index eb96408..7e2370b 100644
--- a/parser/parser.go
+++ b/parser/parser.go
@@ -57,10 +57,28 @@ func ParseFile(filename string, mode Mode, opts ...Option) (*ast.File, error) {
 	return f, nil
 }
 
+type YAMLVersion string
+
+const (
+	YAML10 YAMLVersion = "1.0"
+	YAML11 YAMLVersion = "1.1"
+	YAML12 YAMLVersion = "1.2"
+	YAML13 YAMLVersion = "1.3"
+)
+
+var yamlVersionMap = map[string]YAMLVersion{
+	"1.0": YAML10,
+	"1.1": YAML11,
+	"1.2": YAML12,
+	"1.3": YAML13,
+}
+
 type parser struct {
-	tokens               []*Token
-	pathMap              map[string]ast.Node
-	allowDuplicateMapKey bool
+	tokens                []*Token
+	pathMap               map[string]ast.Node
+	yamlVersion           YAMLVersion
+	allowDuplicateMapKey  bool
+	secondaryTagDirective *ast.DirectiveNode
 }
 
 func newParser(tokens token.Tokens, mode Mode, opts []Option) (*parser, error) {
@@ -122,6 +140,10 @@ func (p *parser) parseDocument(ctx *context, docGroup *TokenGroup) (*ast.Documen
 	if docGroup.Last().Type() == token.DocumentEndType {
 		end = docGroup.Last().RawToken()
 		tokens = tokens[:len(tokens)-1]
+		defer func() {
+			// clear yaml version value if DocumentEnd token (...) is specified.
+			p.yamlVersion = ""
+		}()
 	}
 
 	if len(tokens) == 0 {
@@ -162,6 +184,13 @@ func (p *parser) parseToken(ctx *context, tk *Token) (ast.Node, error) {
 		}
 		ctx.goNext()
 		return node, nil
+	case TokenGroupDirectiveName:
+		node, err := p.parseDirectiveName(ctx.withGroup(tk.Group))
+		if err != nil {
+			return nil, err
+		}
+		ctx.goNext()
+		return node, nil
 	case TokenGroupAnchor:
 		node, err := p.parseAnchor(ctx.withGroup(tk.Group), tk.Group)
 		if err != nil {
@@ -182,6 +211,9 @@ func (p *parser) parseToken(ctx *context, tk *Token) (ast.Node, error) {
 		if err != nil {
 			return nil, err
 		}
+		if _, ok := value.(*ast.AnchorNode); ok {
+			return nil, errors.ErrSyntax("anchors cannot be used consecutively", value.GetToken())
+		}
 		anchor.Value = value
 		return anchor, nil
 	case TokenGroupAlias:
@@ -212,9 +244,9 @@ func (p *parser) parseToken(ctx *context, tk *Token) (ast.Node, error) {
 	case token.TagType:
 		return p.parseTag(ctx)
 	case token.MappingStartType:
-		return p.parseFlowMap(ctx)
+		return p.parseFlowMap(ctx.withFlow(true))
 	case token.SequenceStartType:
-		return p.parseFlowSequence(ctx)
+		return p.parseFlowSequence(ctx.withFlow(true))
 	case token.SequenceEntryType:
 		return p.parseSequence(ctx)
 	case token.SequenceEndType:
@@ -254,6 +286,9 @@ func (p *parser) parseScalarValue(ctx *context, tk *Token) (ast.ScalarNode, erro
 			if err != nil {
 				return nil, err
 			}
+			if _, ok := value.(*ast.AnchorNode); ok {
+				return nil, errors.ErrSyntax("anchors cannot be used consecutively", value.GetToken())
+			}
 			anchor.Value = value
 			return anchor, nil
 		case TokenGroupAlias:
@@ -442,11 +477,18 @@ func (p *parser) parseMap(ctx *context) (*ast.MappingNode, error) {
 		tk = ctx.currentToken()
 	}
 	for tk.Column() == keyTk.Column() {
+		typ := tk.Type()
+		if ctx.isFlow && typ == token.SequenceEndType {
+			// [
+			// key: value
+			// ] <=
+			break
+		}
 		if !p.isMapToken(tk) {
 			return nil, errors.ErrSyntax("non-map value is specified", tk.RawToken())
 		}
 		cm := p.parseHeadComment(ctx)
-		if tk.Type() == token.MappingEndType {
+		if typ == token.MappingEndType {
 			// a: {
 			//  b: c
 			// } <=
@@ -552,7 +594,7 @@ func (p *parser) parseMapKey(ctx *context, g *TokenGroup) (ast.MapKeyNode, error
 		keyText := p.mapKeyText(scalar)
 		keyPath := ctx.withChild(keyText).path
 		key.SetPath(keyPath)
-		if err := p.validateMapKey(key.GetToken(), keyPath); err != nil {
+		if err := p.validateMapKey(ctx, key.GetToken(), keyPath, g.Last()); err != nil {
 			return nil, err
 		}
 		p.pathMap[keyPath] = key
@@ -573,14 +615,14 @@ func (p *parser) parseMapKey(ctx *context, g *TokenGroup) (ast.MapKeyNode, error
 	keyText := p.mapKeyText(key)
 	keyPath := ctx.withChild(keyText).path
 	key.SetPath(keyPath)
-	if err := p.validateMapKey(key.GetToken(), keyPath); err != nil {
+	if err := p.validateMapKey(ctx, key.GetToken(), keyPath, g.Last()); err != nil {
 		return nil, err
 	}
 	p.pathMap[keyPath] = key
 	return key, nil
 }
 
-func (p *parser) validateMapKey(tk *token.Token, keyPath string) error {
+func (p *parser) validateMapKey(ctx *context, tk *token.Token, keyPath string, colonTk *Token) error {
 	if !p.allowDuplicateMapKey {
 		if n, exists := p.pathMap[keyPath]; exists {
 			pos := n.GetToken().Position
@@ -590,29 +632,57 @@ func (p *parser) validateMapKey(tk *token.Token, keyPath string) error {
 			)
 		}
 	}
-	if tk.Type != token.StringType {
+	origin := p.removeLeftWhiteSpace(tk.Origin)
+	if ctx.isFlow {
+		if tk.Type == token.StringType {
+			origin = p.removeRightWhiteSpace(origin)
+			if tk.Position.Line+p.newLineCharacterNum(origin) != colonTk.Line() {
+				return errors.ErrSyntax("map key definition includes an implicit line break", tk)
+			}
+		}
+		return nil
+	}
+	if tk.Type != token.StringType && tk.Type != token.SingleQuoteType && tk.Type != token.DoubleQuoteType {
 		return nil
 	}
-	origin := p.removeLeftSideNewLineCharacter(tk.Origin)
 	if p.existsNewLineCharacter(origin) {
 		return errors.ErrSyntax("unexpected key name", tk)
 	}
 	return nil
 }
 
-func (p *parser) removeLeftSideNewLineCharacter(src string) string {
+func (p *parser) removeLeftWhiteSpace(src string) string {
 	// CR or LF or CRLF
-	return strings.TrimLeft(strings.TrimLeft(strings.TrimLeft(src, "\r"), "\n"), "\r\n")
+	return strings.TrimLeftFunc(src, func(r rune) bool {
+		return r == ' ' || r == '\r' || r == '\n'
+	})
+}
+
+func (p *parser) removeRightWhiteSpace(src string) string {
+	// CR or LF or CRLF
+	return strings.TrimRightFunc(src, func(r rune) bool {
+		return r == ' ' || r == '\r' || r == '\n'
+	})
 }
 
 func (p *parser) existsNewLineCharacter(src string) bool {
-	if strings.Index(src, "\n") > 0 {
-		return true
-	}
-	if strings.Index(src, "\r") > 0 {
-		return true
+	return p.newLineCharacterNum(src) > 0
+}
+
+func (p *parser) newLineCharacterNum(src string) int {
+	var num int
+	for i := 0; i < len(src); i++ {
+		switch src[i] {
+		case '\r':
+			if len(src) > i+1 && src[i+1] == '\n' {
+				i++
+			}
+			num++
+		case '\n':
+			num++
+		}
 	}
-	return false
+	return num
 }
 
 func (p *parser) mapKeyText(n ast.Node) string {
@@ -644,6 +714,15 @@ func (p *parser) parseMapValue(ctx *context, key ast.MapKeyNode, colonTk *Token)
 	keyCol := key.GetToken().Position.Column
 	keyLine := key.GetToken().Position.Line
 
+	if tk.Column() != keyCol && tk.Line() == keyLine && (tk.GroupType() == TokenGroupMapKey || tk.GroupType() == TokenGroupMapKeyValue) {
+		// a: b:
+		//    ^
+		//
+		// a: b: c
+		//    ^
+		return nil, errors.ErrSyntax("mapping value is not allowed in this context", tk.RawToken())
+	}
+
 	if tk.Column() == keyCol && p.isMapToken(tk) {
 		// in this case,
 		// ----
@@ -670,6 +749,17 @@ func (p *parser) parseMapValue(ctx *context, key ast.MapKeyNode, colonTk *Token)
 		return anchor, nil
 	}
 
+	if tk.Column() <= keyCol && tk.GroupType() == TokenGroupAnchorName {
+		// key: <value does not defined>
+		// &anchor
+		return nil, errors.ErrSyntax("anchor is not allowed in this context", tk.RawToken())
+	}
+	if tk.Column() <= keyCol && tk.Type() == token.TagType {
+		// key: <value does not defined>
+		// !!tag
+		return nil, errors.ErrSyntax("tag is not allowed in this context", tk.RawToken())
+	}
+
 	if tk.Column() < keyCol {
 		// in this case,
 		// ----
@@ -700,9 +790,43 @@ func (p *parser) parseMapValue(ctx *context, key ast.MapKeyNode, colonTk *Token)
 	if err != nil {
 		return nil, err
 	}
+	if err := p.validateAnchorValueInMapOrSeq(value, keyCol); err != nil {
+		return nil, err
+	}
 	return value, nil
 }
 
+func (p *parser) validateAnchorValueInMapOrSeq(value ast.Node, col int) error {
+	anchor, ok := value.(*ast.AnchorNode)
+	if !ok {
+		return nil
+	}
+	tag, ok := anchor.Value.(*ast.TagNode)
+	if !ok {
+		return nil
+	}
+	anchorTk := anchor.GetToken()
+	tagTk := tag.GetToken()
+
+	if anchorTk.Position.Line == tagTk.Position.Line {
+		// key:
+		//   &anchor !!tag
+		//
+		// - &anchor !!tag
+		return nil
+	}
+
+	if tagTk.Position.Column <= col {
+		// key: &anchor
+		// !!tag
+		//
+		// - &anchor
+		// !!tag
+		return errors.ErrSyntax("tag is not allowed in this context", tagTk)
+	}
+	return nil
+}
+
 func (p *parser) parseAnchor(ctx *context, g *TokenGroup) (*ast.AnchorNode, error) {
 	anchorNameGroup := g.First().Group
 	anchor, err := p.parseAnchorName(ctx.withGroup(anchorNameGroup))
@@ -718,6 +842,9 @@ func (p *parser) parseAnchor(ctx *context, g *TokenGroup) (*ast.AnchorNode, erro
 	if err != nil {
 		return nil, err
 	}
+	if _, ok := value.(*ast.AnchorNode); ok {
+		return nil, errors.ErrSyntax("anchors cannot be used consecutively", value.GetToken())
+	}
 	anchor.Value = value
 	return anchor, nil
 }
@@ -816,14 +943,26 @@ func (p *parser) parseTag(ctx *context) (*ast.TagNode, error) {
 	ctx.goNext()
 
 	comment := p.parseHeadComment(ctx)
-	value, err := p.parseTagValue(ctx, tagRawTk, ctx.currentToken())
-	if err != nil {
-		return nil, err
+
+	var tagValue ast.Node
+	if p.secondaryTagDirective != nil {
+		value, err := newStringNode(ctx, ctx.currentToken())
+		if err != nil {
+			return nil, err
+		}
+		tagValue = value
+		node.Directive = p.secondaryTagDirective
+	} else {
+		value, err := p.parseTagValue(ctx, tagRawTk, ctx.currentToken())
+		if err != nil {
+			return nil, err
+		}
+		tagValue = value
 	}
-	if err := setHeadComment(comment, value); err != nil {
+	if err := setHeadComment(comment, tagValue); err != nil {
 		return nil, err
 	}
-	node.Value = value
+	node.Value = tagValue
 	return node, nil
 }
 
@@ -837,7 +976,7 @@ func (p *parser) parseTagValue(ctx *context, tagRawTk *token.Token, tk *Token) (
 			return nil, errors.ErrSyntax("could not find map", tk.RawToken())
 		}
 		if tk.Type() == token.MappingStartType {
-			return p.parseFlowMap(ctx)
+			return p.parseFlowMap(ctx.withFlow(true))
 		}
 		return p.parseMap(ctx)
 	case token.IntegerTag, token.FloatTag, token.StringTag, token.BinaryTag, token.TimestampTag, token.BooleanTag, token.NullTag:
@@ -854,13 +993,10 @@ func (p *parser) parseTagValue(ctx *context, tagRawTk *token.Token, tk *Token) (
 		return scalar, nil
 	case token.SequenceTag, token.OrderedMapTag:
 		if tk.Type() == token.SequenceStartType {
-			return p.parseFlowSequence(ctx)
+			return p.parseFlowSequence(ctx.withFlow(true))
 		}
 		return p.parseSequence(ctx)
 	}
-	if strings.HasPrefix(tagRawTk.Value, "!!") {
-		return nil, errors.ErrSyntax(fmt.Sprintf("unknown secondary tag name %q specified", tagRawTk.Value), tagRawTk)
-	}
 	return p.parseToken(ctx, tk)
 }
 
@@ -896,7 +1032,7 @@ func (p *parser) parseFlowSequence(ctx *context) (*ast.SequenceNode, error) {
 			break
 		}
 
-		value, err := p.parseToken(ctx.withIndex(uint(len(node.Values))).withFlow(true), ctx.currentToken())
+		value, err := p.parseToken(ctx.withIndex(uint(len(node.Values))), ctx.currentToken())
 		if err != nil {
 			return nil, err
 		}
@@ -923,17 +1059,7 @@ func (p *parser) parseSequence(ctx *context) (*ast.SequenceNode, error) {
 		comment := p.parseHeadComment(ctx)
 		ctx.goNext() // skip sequence entry token
 
-		valueTk := ctx.currentToken()
-		if valueTk == nil {
-			node, err := newNullNode(ctx, ctx.createNullToken(seqTk))
-			if err != nil {
-				return nil, err
-			}
-			seqNode.Values = append(seqNode.Values, node)
-			break
-		}
-
-		value, err := p.parseToken(ctx.withIndex(uint(len(seqNode.Values))), valueTk)
+		value, err := p.parseSequenceValue(ctx.withIndex(uint(len(seqNode.Values))), seqTk)
 		if err != nil {
 			return nil, err
 		}
@@ -959,21 +1085,175 @@ func (p *parser) parseSequence(ctx *context) (*ast.SequenceNode, error) {
 	return seqNode, nil
 }
 
+func (p *parser) parseSequenceValue(ctx *context, seqTk *Token) (ast.Node, error) {
+	tk := ctx.currentToken()
+	if tk == nil {
+		return newNullNode(ctx, ctx.insertNullToken(seqTk))
+	}
+
+	if ctx.isComment() {
+		tk = ctx.nextNotCommentToken()
+	}
+	seqCol := seqTk.Column()
+	seqLine := seqTk.Line()
+
+	if tk.Column() == seqCol && tk.Type() == token.SequenceEntryType {
+		// in this case,
+		// ----
+		// - <value does not defined>
+		// -
+		return newNullNode(ctx, ctx.insertNullToken(seqTk))
+	}
+
+	if tk.Line() == seqLine && tk.GroupType() == TokenGroupAnchorName &&
+		ctx.nextToken().Column() == seqCol && ctx.nextToken().Type() == token.SequenceEntryType {
+		// in this case,
+		// ----
+		// - &anchor
+		// -
+		group := &TokenGroup{
+			Type:   TokenGroupAnchor,
+			Tokens: []*Token{tk, ctx.createNullToken(tk)},
+		}
+		anchor, err := p.parseAnchor(ctx.withGroup(group), group)
+		if err != nil {
+			return nil, err
+		}
+		ctx.goNext()
+		return anchor, nil
+	}
+
+	if tk.Column() <= seqCol && tk.GroupType() == TokenGroupAnchorName {
+		// - <value does not defined>
+		// &anchor
+		return nil, errors.ErrSyntax("anchor is not allowed in this sequence context", tk.RawToken())
+	}
+	if tk.Column() <= seqCol && tk.Type() == token.TagType {
+		// - <value does not defined>
+		// !!tag
+		return nil, errors.ErrSyntax("tag is not allowed in this sequence context", tk.RawToken())
+	}
+
+	if tk.Column() < seqCol {
+		// in this case,
+		// ----
+		//   - <value does not defined>
+		// next
+		return newNullNode(ctx, ctx.insertNullToken(seqTk))
+	}
+
+	if tk.Line() == seqLine && tk.GroupType() == TokenGroupAnchorName &&
+		ctx.nextToken().Column() < seqCol {
+		// in this case,
+		// ----
+		//   - &anchor
+		// next
+		group := &TokenGroup{
+			Type:   TokenGroupAnchor,
+			Tokens: []*Token{tk, ctx.createNullToken(tk)},
+		}
+		anchor, err := p.parseAnchor(ctx.withGroup(group), group)
+		if err != nil {
+			return nil, err
+		}
+		ctx.goNext()
+		return anchor, nil
+	}
+
+	value, err := p.parseToken(ctx, ctx.currentToken())
+	if err != nil {
+		return nil, err
+	}
+	if err := p.validateAnchorValueInMapOrSeq(value, seqCol); err != nil {
+		return nil, err
+	}
+	return value, nil
+}
+
 func (p *parser) parseDirective(ctx *context, g *TokenGroup) (*ast.DirectiveNode, error) {
-	node, err := newDirectiveNode(ctx, g.First())
+	directiveNameGroup := g.First().Group
+	directive, err := p.parseDirectiveName(ctx.withGroup(directiveNameGroup))
+	if err != nil {
+		return nil, err
+	}
+
+	switch directive.Name.String() {
+	case "YAML":
+		if len(g.Tokens) != 2 {
+			return nil, errors.ErrSyntax("unexpected format YAML directive", g.First().RawToken())
+		}
+		valueTk := g.Tokens[1]
+		valueRawTk := valueTk.RawToken()
+		value := valueRawTk.Value
+		ver, exists := yamlVersionMap[value]
+		if !exists {
+			return nil, errors.ErrSyntax(fmt.Sprintf("unknown YAML version %q", value), valueRawTk)
+		}
+		if p.yamlVersion != "" {
+			return nil, errors.ErrSyntax("YAML version has already been specified", valueRawTk)
+		}
+		p.yamlVersion = ver
+		versionNode, err := newStringNode(ctx, valueTk)
+		if err != nil {
+			return nil, err
+		}
+		directive.Values = append(directive.Values, versionNode)
+	case "TAG":
+		if len(g.Tokens) != 3 {
+			return nil, errors.ErrSyntax("unexpected format TAG directive", g.First().RawToken())
+		}
+		tagKey, err := newStringNode(ctx, g.Tokens[1])
+		if err != nil {
+			return nil, err
+		}
+		if tagKey.Value == "!!" {
+			p.secondaryTagDirective = directive
+		}
+		tagValue, err := newStringNode(ctx, g.Tokens[2])
+		if err != nil {
+			return nil, err
+		}
+		directive.Values = append(directive.Values, tagKey, tagValue)
+	default:
+		if len(g.Tokens) > 1 {
+			for _, tk := range g.Tokens[1:] {
+				value, err := newStringNode(ctx, tk)
+				if err != nil {
+					return nil, err
+				}
+				directive.Values = append(directive.Values, value)
+			}
+		}
+	}
+	return directive, nil
+}
+
+func (p *parser) parseDirectiveName(ctx *context) (*ast.DirectiveNode, error) {
+	directive, err := newDirectiveNode(ctx, ctx.currentToken())
 	if err != nil {
 		return nil, err
 	}
-	value, err := p.parseToken(ctx, g.Last())
+	ctx.goNext()
+	if ctx.isTokenNotFound() {
+		return nil, errors.ErrSyntax("could not find directive value", directive.GetToken())
+	}
+
+	directiveName, err := p.parseScalarValue(ctx, ctx.currentToken())
 	if err != nil {
 		return nil, err
 	}
-	node.Value = value
-	return node, nil
+	if directiveName == nil {
+		return nil, errors.ErrSyntax("unexpected directive. directive name is not scalar value", ctx.currentToken().RawToken())
+	}
+	directive.Name = directiveName
+	return directive, nil
 }
 
 func (p *parser) parseComment(ctx *context) (ast.Node, error) {
 	cm := p.parseHeadComment(ctx)
+	if ctx.isTokenNotFound() {
+		return cm, nil
+	}
 	node, err := p.parseToken(ctx, ctx.currentToken())
 	if err != nil {
 		return nil, err
diff --git a/parser/token.go b/parser/token.go
index 9897492..3d1c06f 100644
--- a/parser/token.go
+++ b/parser/token.go
@@ -3,6 +3,7 @@ package parser
 import (
 	"fmt"
 	"os"
+	"strings"
 
 	"github.com/goccy/go-yaml/internal/errors"
 	"github.com/goccy/go-yaml/token"
@@ -13,6 +14,7 @@ type TokenGroupType int
 const (
 	TokenGroupNone TokenGroupType = iota
 	TokenGroupDirective
+	TokenGroupDirectiveName
 	TokenGroupDocument
 	TokenGroupDocumentBody
 	TokenGroupAnchor
@@ -31,6 +33,8 @@ func (t TokenGroupType) String() string {
 		return "none"
 	case TokenGroupDirective:
 		return "directive"
+	case TokenGroupDirectiveName:
+		return "directive_name"
 	case TokenGroupDocument:
 		return "document"
 	case TokenGroupDocumentBody:
@@ -353,8 +357,37 @@ func createScalarTagTokenGroups(tokens []*Token) []*Token {
 			continue
 		}
 		tag := tk.RawToken()
-		switch token.ReservedTagKeyword(tag.Value) {
-		case token.IntegerTag, token.FloatTag, token.StringTag, token.BinaryTag, token.TimestampTag, token.BooleanTag, token.NullTag:
+		if strings.HasPrefix(tag.Value, "!!") {
+			// secondary tag.
+			switch token.ReservedTagKeyword(tag.Value) {
+			case token.IntegerTag, token.FloatTag, token.StringTag, token.BinaryTag, token.TimestampTag, token.BooleanTag, token.NullTag:
+				if len(tokens) <= i+1 {
+					ret = append(ret, tk)
+					continue
+				}
+				if tk.Line() != tokens[i+1].Line() {
+					ret = append(ret, tk)
+					continue
+				}
+				if tokens[i+1].GroupType() == TokenGroupAnchorName {
+					ret = append(ret, tk)
+					continue
+				}
+				if isScalarType(tokens[i+1]) {
+					ret = append(ret, &Token{
+						Group: &TokenGroup{
+							Type:   TokenGroupScalarTag,
+							Tokens: []*Token{tk, tokens[i+1]},
+						},
+					})
+					i++
+				} else {
+					ret = append(ret, tk)
+				}
+			default:
+				ret = append(ret, tk)
+			}
+		} else {
 			if len(tokens) <= i+1 {
 				ret = append(ret, tk)
 				continue
@@ -367,19 +400,13 @@ func createScalarTagTokenGroups(tokens []*Token) []*Token {
 				ret = append(ret, tk)
 				continue
 			}
-			if isScalarType(tokens[i+1]) {
-				ret = append(ret, &Token{
-					Group: &TokenGroup{
-						Type:   TokenGroupScalarTag,
-						Tokens: []*Token{tk, tokens[i+1]},
-					},
-				})
-				i++
-			} else {
-				ret = append(ret, tk)
-			}
-		default:
-			ret = append(ret, tk)
+			ret = append(ret, &Token{
+				Group: &TokenGroup{
+					Type:   TokenGroupScalarTag,
+					Tokens: []*Token{tk, tokens[i+1]},
+				},
+			})
+			i++
 		}
 	}
 	return ret
@@ -522,22 +549,34 @@ func createDirectiveTokenGroups(tokens []*Token) ([]*Token, error) {
 			if i+1 >= len(tokens) {
 				return nil, errors.ErrSyntax("undefined directive value", tk.RawToken())
 			}
-			if i+2 >= len(tokens) {
-				return nil, errors.ErrSyntax("unexpected directive value. document not started", tk.RawToken())
-			}
-			if tokens[i+2].Type() != token.DocumentHeaderType {
-				return nil, errors.ErrSyntax("unexpected directive value. document not started", tk.RawToken())
-			}
-			if tk.Line() != tokens[i+1].Line() {
-				return nil, errors.ErrSyntax("undefined directive value", tk.RawToken())
-			}
-			ret = append(ret, &Token{
+			directiveName := &Token{
 				Group: &TokenGroup{
-					Type:   TokenGroupDirective,
+					Type:   TokenGroupDirectiveName,
 					Tokens: []*Token{tk, tokens[i+1]},
 				},
-			})
+			}
 			i++
+			var valueTks []*Token
+			for j := i + 1; j < len(tokens); j++ {
+				if tokens[j].Line() != tk.Line() {
+					break
+				}
+				valueTks = append(valueTks, tokens[j])
+				i++
+			}
+			if i+1 >= len(tokens) || tokens[i+1].Type() != token.DocumentHeaderType {
+				return nil, errors.ErrSyntax("unexpected directive value. document not started", tk.RawToken())
+			}
+			if len(valueTks) != 0 {
+				ret = append(ret, &Token{
+					Group: &TokenGroup{
+						Type:   TokenGroupDirective,
+						Tokens: append([]*Token{directiveName}, valueTks...),
+					},
+				})
+			} else {
+				ret = append(ret, directiveName)
+			}
 		default:
 			ret = append(ret, tk)
 		}
@@ -573,6 +612,16 @@ func createDocumentTokens(tokens []*Token) ([]*Token, error) {
 					},
 				}), nil
 			}
+			if tokens[i].Line() == tokens[i+1].Line() {
+				switch tokens[i+1].GroupType() {
+				case TokenGroupMapKey, TokenGroupMapKeyValue:
+					return nil, errors.ErrSyntax("value cannot be placed after document separator", tokens[i+1].RawToken())
+				}
+				switch tokens[i+1].Type() {
+				case token.SequenceEntryType:
+					return nil, errors.ErrSyntax("value cannot be placed after document separator", tokens[i+1].RawToken())
+				}
+			}
 			tks, err := createDocumentTokens(tokens[i+1:])
 			if err != nil {
 				return nil, err
@@ -620,6 +669,10 @@ func createDocumentTokens(tokens []*Token) ([]*Token, error) {
 }
 
 func isScalarType(tk *Token) bool {
+	switch tk.GroupType() {
+	case TokenGroupMapKey, TokenGroupMapKeyValue:
+		return false
+	}
 	typ := tk.Type()
 	return typ == token.AnchorType ||
 		typ == token.AliasType ||
diff --git a/scanner/context.go b/scanner/context.go
index 6d5e7eb..2ca1a13 100644
--- a/scanner/context.go
+++ b/scanner/context.go
@@ -1,7 +1,7 @@
 package scanner
 
 import (
-	"fmt"
+	"errors"
 	"strconv"
 	"strings"
 	"sync"
@@ -26,6 +26,7 @@ type Context struct {
 	docFirstLineIndentColumn int
 	docPrevLineIndentColumn  int
 	docLineIndentColumn      int
+	docSpaceOnlyIndentColumn int
 	docFoldedNewLine         bool
 }
 
@@ -63,6 +64,7 @@ func (c *Context) clear() {
 	c.docFirstLineIndentColumn = 0
 	c.docLineIndentColumn = 0
 	c.docPrevLineIndentColumn = 0
+	c.docSpaceOnlyIndentColumn = 0
 	c.docFoldedNewLine = false
 }
 
@@ -93,6 +95,7 @@ func (c *Context) breakDocument() {
 	c.docFirstLineIndentColumn = 0
 	c.docLineIndentColumn = 0
 	c.docPrevLineIndentColumn = 0
+	c.docSpaceOnlyIndentColumn = 0
 	c.docFoldedNewLine = false
 }
 
@@ -122,12 +125,29 @@ func (c *Context) updateDocumentLineIndentColumn(column int) {
 	}
 }
 
+func (c *Context) updateSpaceOnlyIndentColumn(column int) {
+	if c.docFirstLineIndentColumn != 0 {
+		return
+	}
+	c.docSpaceOnlyIndentColumn = column
+}
+
+func (c *Context) validateDocumentLineIndentAfterSpaceOnly(column int) error {
+	if c.docFirstLineIndentColumn != 0 {
+		return nil
+	}
+	if c.docSpaceOnlyIndentColumn > column {
+		return errors.New("invalid number of indent is specified after space only")
+	}
+	return nil
+}
+
 func (c *Context) validateDocumentLineIndentColumn() error {
 	if c.docFirstLineIndentColumnByDocOpt() == 0 {
 		return nil
 	}
 	if c.docFirstLineIndentColumn > c.docLineIndentColumn {
-		return fmt.Errorf("invalid number of indent is specified in the document header")
+		return errors.New("invalid number of indent is specified in the document header")
 	}
 	return nil
 }
@@ -352,10 +372,25 @@ func (c *Context) bufferedToken(pos *token.Position) *token.Token {
 	} else {
 		tk = token.New(string(source), string(c.obuf), pos)
 	}
+	c.setTokenTypeByPrevTag(tk)
 	c.resetBuffer()
 	return tk
 }
 
+func (c *Context) setTokenTypeByPrevTag(tk *token.Token) {
+	lastTk := c.lastToken()
+	if lastTk == nil {
+		return
+	}
+	if lastTk.Type != token.TagType {
+		return
+	}
+	tag := token.ReservedTagKeyword(lastTk.Value)
+	if _, exists := token.ReservedTagKeywordMap[tag]; !exists {
+		tk.Type = token.StringType
+	}
+}
+
 func (c *Context) lastToken() *token.Token {
 	if len(c.tokens) != 0 {
 		return c.tokens[len(c.tokens)-1]
diff --git a/scanner/scanner.go b/scanner/scanner.go
index 501534e..796d0ba 100644
--- a/scanner/scanner.go
+++ b/scanner/scanner.go
@@ -255,6 +255,11 @@ func (s *Scanner) scanSingleQuote(ctx *Context) (*token.Token, error) {
 			isFirstLineChar = true
 			isNewLine = true
 			s.progressLine(ctx)
+			if idx+1 < size {
+				if err := s.validateDocumentSeparatorMarker(ctx, src[idx+1:]); err != nil {
+					return nil, err
+				}
+			}
 			continue
 		} else if isFirstLineChar && (c == ' ' || c == '\t') {
 			continue
@@ -262,8 +267,7 @@ func (s *Scanner) scanSingleQuote(ctx *Context) (*token.Token, error) {
 			value = append(value, c)
 			isFirstLineChar = false
 			continue
-		}
-		if idx+1 < len(ctx.src) && ctx.src[idx+1] == '\'' {
+		} else if idx+1 < len(ctx.src) && ctx.src[idx+1] == '\'' {
 			// '' handle as ' character
 			value = append(value, c)
 			ctx.addOriginBuf(c)
@@ -339,6 +343,11 @@ func (s *Scanner) scanDoubleQuote(ctx *Context) (*token.Token, error) {
 			isFirstLineChar = true
 			isNewLine = true
 			s.progressLine(ctx)
+			if idx+1 < size {
+				if err := s.validateDocumentSeparatorMarker(ctx, src[idx+1:]); err != nil {
+					return nil, err
+				}
+			}
 			continue
 		} else if isFirstLineChar && (c == ' ' || c == '\t') {
 			continue
@@ -351,54 +360,74 @@ func (s *Scanner) scanDoubleQuote(ctx *Context) (*token.Token, error) {
 			nextChar := src[idx+1]
 			progress := 0
 			switch nextChar {
-			case 'b':
+			case '0':
 				progress = 1
 				ctx.addOriginBuf(nextChar)
-				value = append(value, '\b')
-			case 'e':
+				value = append(value, 0x00)
+			case 'a':
 				progress = 1
 				ctx.addOriginBuf(nextChar)
-				value = append(value, '\x1B')
-			case 'f':
+				value = append(value, 0x07)
+			case 'b':
 				progress = 1
 				ctx.addOriginBuf(nextChar)
-				value = append(value, '\f')
+				value = append(value, 0x08)
+			case 't':
+				progress = 1
+				ctx.addOriginBuf(nextChar)
+				value = append(value, 0x09)
 			case 'n':
 				progress = 1
 				ctx.addOriginBuf(nextChar)
-				value = append(value, '\n')
+				value = append(value, 0x0A)
+			case 'v':
+				progress = 1
+				ctx.addOriginBuf(nextChar)
+				value = append(value, 0x0B)
+			case 'f':
+				progress = 1
+				ctx.addOriginBuf(nextChar)
+				value = append(value, 0x0C)
 			case 'r':
 				progress = 1
 				ctx.addOriginBuf(nextChar)
-				value = append(value, '\r')
-			case 't':
+				value = append(value, 0x0D)
+			case 'e':
 				progress = 1
 				ctx.addOriginBuf(nextChar)
-				value = append(value, '\t')
-			case 'v':
+				value = append(value, 0x1B)
+			case ' ':
+				progress = 1
+				ctx.addOriginBuf(nextChar)
+				value = append(value, 0x20)
+			case '"':
 				progress = 1
 				ctx.addOriginBuf(nextChar)
-				value = append(value, '\v')
-			case 'L': // LS (#x2028)
+				value = append(value, 0x22)
+			case '/':
 				progress = 1
 				ctx.addOriginBuf(nextChar)
-				value = append(value, []rune{'\xE2', '\x80', '\xA8'}...)
-			case 'N': // NEL (#x85)
+				value = append(value, 0x2F)
+			case '\\':
 				progress = 1
 				ctx.addOriginBuf(nextChar)
-				value = append(value, []rune{'\xC2', '\x85'}...)
-			case 'P': // PS (#x2029)
+				value = append(value, 0x5C)
+			case 'N':
 				progress = 1
 				ctx.addOriginBuf(nextChar)
-				value = append(value, []rune{'\xE2', '\x80', '\xA9'}...)
-			case '_': // #xA0
+				value = append(value, 0x85)
+			case '_':
 				progress = 1
 				ctx.addOriginBuf(nextChar)
-				value = append(value, []rune{'\xC2', '\xA0'}...)
-			case '"':
+				value = append(value, 0xA0)
+			case 'L':
 				progress = 1
 				ctx.addOriginBuf(nextChar)
-				value = append(value, nextChar)
+				value = append(value, 0x2028)
+			case 'P':
+				progress = 1
+				ctx.addOriginBuf(nextChar)
+				value = append(value, 0x2029)
 			case 'x':
 				if idx+3 >= size {
 					progress = 1
@@ -410,29 +439,67 @@ func (s *Scanner) scanDoubleQuote(ctx *Context) (*token.Token, error) {
 					value = append(value, rune(codeNum))
 				}
 			case 'u':
+				// \u0000 style must have 5 characters at least.
 				if idx+5 >= size {
-					progress = 1
-					ctx.addOriginBuf(nextChar)
-					value = append(value, nextChar)
-				} else {
-					progress = 5
-					codeNum := hexRunesToInt(src[idx+2 : idx+progress+1])
-					value = append(value, rune(codeNum))
+					return nil, ErrInvalidToken(
+						token.Invalid(
+							"not enough length for escaped UTF-16 character",
+							string(ctx.obuf), s.pos(),
+						),
+					)
+				}
+				progress = 5
+				codeNum := hexRunesToInt(src[idx+2 : idx+6])
+
+				// handle surrogate pairs.
+				if codeNum >= 0xD800 && codeNum <= 0xDBFF {
+					high := codeNum
+
+					// \u0000\u0000 style must have 11 characters at least.
+					if idx+11 >= size {
+						return nil, ErrInvalidToken(
+							token.Invalid(
+								"not enough length for escaped UTF-16 surrogate pair",
+								string(ctx.obuf), s.pos(),
+							),
+						)
+					}
+
+					if src[idx+6] != '\\' || src[idx+7] != 'u' {
+						return nil, ErrInvalidToken(
+							token.Invalid(
+								"found unexpected character after high surrogate for UTF-16 surrogate pair",
+								string(ctx.obuf), s.pos(),
+							),
+						)
+					}
+
+					low := hexRunesToInt(src[idx+8 : idx+12])
+					if low < 0xDC00 || low > 0xDFFF {
+						return nil, ErrInvalidToken(
+							token.Invalid(
+								"found unexpected low surrogate after high surrogate",
+								string(ctx.obuf), s.pos(),
+							),
+						)
+					}
+					codeNum = ((high - 0xD800) * 0x400) + (low - 0xDC00) + 0x10000
+					progress += 6
 				}
+				value = append(value, rune(codeNum))
 			case 'U':
+				// \U00000000 style must have 9 characters at least.
 				if idx+9 >= size {
-					progress = 1
-					ctx.addOriginBuf(nextChar)
-					value = append(value, nextChar)
-				} else {
-					progress = 9
-					codeNum := hexRunesToInt(src[idx+2 : idx+progress+1])
-					value = append(value, rune(codeNum))
+					return nil, ErrInvalidToken(
+						token.Invalid(
+							"not enough length for escaped UTF-32 character",
+							string(ctx.obuf), s.pos(),
+						),
+					)
 				}
-			case '\\':
-				progress = 1
-				ctx.addOriginBuf(nextChar)
-				value = append(value, c)
+				progress = 9
+				codeNum := hexRunesToInt(src[idx+2 : idx+10])
+				value = append(value, rune(codeNum))
 			case '\n':
 				isFirstLineChar = true
 				isNewLine = true
@@ -445,10 +512,14 @@ func (s *Scanner) scanDoubleQuote(ctx *Context) (*token.Token, error) {
 				progress = 1
 				ctx.addOriginBuf(nextChar)
 				value = append(value, nextChar)
-			case ' ':
-				// skip escape character.
 			default:
-				value = append(value, c)
+				s.progressColumn(ctx, 1)
+				return nil, ErrInvalidToken(
+					token.Invalid(
+						fmt.Sprintf("found unknown escape character %q", nextChar),
+						string(ctx.obuf), s.pos(),
+					),
+				)
 			}
 			idx += progress
 			s.progressColumn(ctx, progress)
@@ -495,6 +566,30 @@ func (s *Scanner) scanDoubleQuote(ctx *Context) (*token.Token, error) {
 	)
 }
 
+func (s *Scanner) validateDocumentSeparatorMarker(ctx *Context, src []rune) error {
+	if s.foundDocumentSeparatorMarker(src) {
+		return ErrInvalidToken(
+			token.Invalid("found unexpected document separator", string(ctx.obuf), s.pos()),
+		)
+	}
+	return nil
+}
+
+func (s *Scanner) foundDocumentSeparatorMarker(src []rune) bool {
+	if len(src) < 3 {
+		return false
+	}
+	var marker string
+	if len(src) == 3 {
+		marker = string(src)
+	} else {
+		marker = strings.TrimRightFunc(string(src[:4]), func(r rune) bool {
+			return r == ' ' || r == '\t' || r == '\n' || r == '\r'
+		})
+	}
+	return marker == "---" || marker == "..."
+}
+
 func (s *Scanner) scanQuote(ctx *Context, ch rune) (bool, error) {
 	if ctx.existsBuffer() {
 		return false, nil
@@ -520,7 +615,7 @@ func (s *Scanner) scanWhiteSpace(ctx *Context) bool {
 	if ctx.isDocument() {
 		return false
 	}
-	if !s.isAnchor && !s.isAlias && !s.isFirstCharAtLine {
+	if !s.isAnchor && !s.isDirective && !s.isAlias && !s.isFirstCharAtLine {
 		return false
 	}
 
@@ -529,6 +624,12 @@ func (s *Scanner) scanWhiteSpace(ctx *Context) bool {
 		ctx.addOriginBuf(' ')
 		return true
 	}
+	if s.isDirective {
+		s.addBufferedTokenIfExists(ctx)
+		s.progressColumn(ctx, 1)
+		ctx.addOriginBuf(' ')
+		return true
+	}
 
 	s.addBufferedTokenIfExists(ctx)
 	s.isAnchor = false
@@ -561,7 +662,7 @@ func (s *Scanner) isMergeKey(ctx *Context) bool {
 }
 
 func (s *Scanner) scanTag(ctx *Context) bool {
-	if ctx.existsBuffer() {
+	if ctx.existsBuffer() || s.isDirective {
 		return false
 	}
 
@@ -606,8 +707,11 @@ func (s *Scanner) scanTag(ctx *Context) bool {
 }
 
 func (s *Scanner) scanComment(ctx *Context) bool {
-	if ctx.existsBuffer() && (ctx.previousChar() != ' ' && ctx.previousChar() != '\t') {
-		return false
+	if ctx.existsBuffer() {
+		c := ctx.previousChar()
+		if c != ' ' && c != '\t' && !s.isNewLineChar(c) {
+			return false
+		}
 	}
 
 	s.addBufferedTokenIfExists(ctx)
@@ -616,19 +720,19 @@ func (s *Scanner) scanComment(ctx *Context) bool {
 
 	for idx, c := range ctx.src[ctx.idx:] {
 		ctx.addOriginBuf(c)
-		switch c {
-		case '\n', '\r':
-			if ctx.previousChar() == '\\' {
-				continue
-			}
-			value := ctx.source(ctx.idx, ctx.idx+idx)
-			progress := len([]rune(value))
-			ctx.addToken(token.Comment(value, string(ctx.obuf), s.pos()))
-			s.progressColumn(ctx, progress)
-			s.progressLine(ctx)
-			ctx.clear()
-			return true
+		if !s.isNewLineChar(c) {
+			continue
 		}
+		if ctx.previousChar() == '\\' {
+			continue
+		}
+		value := ctx.source(ctx.idx, ctx.idx+idx)
+		progress := len([]rune(value))
+		ctx.addToken(token.Comment(value, string(ctx.obuf), s.pos()))
+		s.progressColumn(ctx, progress)
+		s.progressLine(ctx)
+		ctx.clear()
+		return true
 	}
 	// document ends with comment.
 	value := string(ctx.src[ctx.idx:])
@@ -660,8 +764,17 @@ func (s *Scanner) scanDocument(ctx *Context, c rune) error {
 		s.progressColumn(ctx, 1)
 	} else if s.isNewLineChar(c) {
 		ctx.addBuf(c)
+		ctx.updateSpaceOnlyIndentColumn(s.column - 1)
 		ctx.updateDocumentNewLineState()
 		s.progressLine(ctx)
+		if ctx.next() {
+			if s.foundDocumentSeparatorMarker(ctx.src[ctx.idx:]) {
+				value := ctx.bufferedSrc()
+				ctx.addToken(token.String(string(value), string(ctx.obuf), s.pos()))
+				ctx.clear()
+				s.breakDocument(ctx)
+			}
+		}
 	} else if s.isFirstCharAtLine && c == ' ' {
 		ctx.addDocumentIndent(s.column)
 		s.progressColumn(ctx, 1)
@@ -675,6 +788,11 @@ func (s *Scanner) scanDocument(ctx *Context, c rune) error {
 		s.progressColumn(ctx, 1)
 		return err
 	} else {
+		if err := ctx.validateDocumentLineIndentAfterSpaceOnly(s.column); err != nil {
+			invalidTk := token.Invalid(err.Error(), string(ctx.obuf), s.pos())
+			s.progressColumn(ctx, 1)
+			return ErrInvalidToken(invalidTk)
+		}
 		ctx.updateDocumentLineIndentColumn(s.column)
 		if ctx.docFirstLineIndentColumn > 0 {
 			s.lastDelimColumn = ctx.docFirstLineIndentColumn - 1
@@ -719,7 +837,7 @@ func (s *Scanner) scanNewLine(ctx *Context, c rune) {
 
 	if ctx.isEOS() {
 		s.addBufferedTokenIfExists(ctx)
-	} else if s.isAnchor || s.isAlias {
+	} else if s.isAnchor || s.isAlias || s.isDirective {
 		s.addBufferedTokenIfExists(ctx)
 	}
 	if ctx.existsBuffer() && s.isFirstCharAtLine {
@@ -822,6 +940,10 @@ func (s *Scanner) scanMapDelim(ctx *Context) (bool, error) {
 	if s.startedFlowMapNum <= 0 && nc != ' ' && nc != '\t' && !s.isNewLineChar(nc) && !ctx.isNextEOS() {
 		return false, nil
 	}
+	if s.startedFlowMapNum > 0 && nc == '/' {
+		// like http://
+		return false, nil
+	}
 
 	if strings.HasPrefix(strings.TrimPrefix(string(ctx.obuf), " "), "\t") && !strings.HasPrefix(string(ctx.buf), "\t") {
 		invalidTk := token.Invalid("tab character cannot use as a map key directly", string(ctx.obuf), s.pos())
@@ -883,6 +1005,7 @@ func (s *Scanner) scanDocumentEnd(ctx *Context) bool {
 		return false
 	}
 
+	s.addBufferedTokenIfExists(ctx)
 	ctx.addToken(token.DocumentEnd(string(ctx.obuf)+"...", s.pos()))
 	s.progressColumn(ctx, 3)
 	ctx.clear()
@@ -960,6 +1083,7 @@ func (s *Scanner) validateDocumentHeaderOption(opt string) error {
 	if len(opt) == 0 {
 		return nil
 	}
+	orgOpt := opt
 	opt = strings.TrimPrefix(opt, "-")
 	opt = strings.TrimPrefix(opt, "+")
 	opt = strings.TrimSuffix(opt, "-")
@@ -967,8 +1091,15 @@ func (s *Scanner) validateDocumentHeaderOption(opt string) error {
 	if len(opt) == 0 {
 		return nil
 	}
-	if _, err := strconv.ParseInt(opt, 10, 64); err != nil {
-		return fmt.Errorf("invalid header option: %q", opt)
+	if opt == "0" {
+		return fmt.Errorf("invalid header option: %q", orgOpt)
+	}
+	i, err := strconv.ParseInt(opt, 10, 64)
+	if err != nil {
+		return fmt.Errorf("invalid header option: %q", orgOpt)
+	}
+	if i > 9 {
+		return fmt.Errorf("invalid header option: %q", orgOpt)
 	}
 	return nil
 }
@@ -977,64 +1108,59 @@ func (s *Scanner) scanDocumentHeaderOption(ctx *Context) error {
 	header := ctx.currentChar()
 	ctx.addOriginBuf(header)
 	s.progress(ctx, 1) // skip '|' or '>' character
+
+	var progress int
 	for idx, c := range ctx.src[ctx.idx:] {
-		progress := idx
+		progress = idx
 		ctx.addOriginBuf(c)
-		switch c {
-		case '\n', '\r':
-			value := strings.TrimRight(ctx.source(ctx.idx, ctx.idx+idx), " ")
-			commentValueIndex := strings.Index(value, "#")
-			opt := value
-			if commentValueIndex > 0 {
-				opt = value[:commentValueIndex]
-			}
-			opt = strings.TrimRightFunc(opt, func(r rune) bool {
-				return r == ' ' || r == '\t'
-			})
-			if len(opt) != 0 {
-				if err := s.validateDocumentHeaderOption(opt); err != nil {
-					invalidTk := token.Invalid(err.Error(), string(ctx.obuf), s.pos())
-					s.progressColumn(ctx, progress)
-					return ErrInvalidToken(invalidTk)
-				}
-			}
-			if s.column == 1 {
-				s.lastDelimColumn = 1
-			}
-
-			commentIndex := strings.Index(string(ctx.obuf), "#")
-			headerBuf := string(ctx.obuf)
-			if commentIndex > 0 {
-				headerBuf = headerBuf[:commentIndex]
-			}
-			switch header {
-			case '|':
-				ctx.addToken(token.Literal("|"+opt, headerBuf, s.pos()))
-				ctx.isLiteral = true
-			case '>':
-				ctx.addToken(token.Folded(">"+opt, headerBuf, s.pos()))
-				ctx.isFolded = true
-			}
-			if commentIndex > 0 {
-				comment := string(value[commentValueIndex+1:])
-				s.offset += len(headerBuf)
-				s.column += len(headerBuf)
-				ctx.addToken(token.Comment(comment, string(ctx.obuf[len(headerBuf):]), s.pos()))
-			}
-			s.indentState = IndentStateKeep
-			ctx.resetBuffer()
-			ctx.docOpt = opt
+		if s.isNewLineChar(c) {
+			break
+		}
+	}
+	value := strings.TrimRight(ctx.source(ctx.idx, ctx.idx+progress), " ")
+	commentValueIndex := strings.Index(value, "#")
+	opt := value
+	if commentValueIndex > 0 {
+		opt = value[:commentValueIndex]
+	}
+	opt = strings.TrimRightFunc(opt, func(r rune) bool {
+		return r == ' ' || r == '\t'
+	})
+	if len(opt) != 0 {
+		if err := s.validateDocumentHeaderOption(opt); err != nil {
+			invalidTk := token.Invalid(err.Error(), string(ctx.obuf), s.pos())
 			s.progressColumn(ctx, progress)
-			return nil
+			return ErrInvalidToken(invalidTk)
 		}
 	}
-	text := string(ctx.src[ctx.idx:])
-	invalidTk := token.Invalid(
-		fmt.Sprintf("invalid document header: %q", text),
-		string(ctx.obuf), s.pos(),
-	)
-	s.progressColumn(ctx, len(text))
-	return ErrInvalidToken(invalidTk)
+	if s.column == 1 {
+		s.lastDelimColumn = 1
+	}
+
+	commentIndex := strings.Index(string(ctx.obuf), "#")
+	headerBuf := string(ctx.obuf)
+	if commentIndex > 0 {
+		headerBuf = headerBuf[:commentIndex]
+	}
+	switch header {
+	case '|':
+		ctx.addToken(token.Literal("|"+opt, headerBuf, s.pos()))
+		ctx.isLiteral = true
+	case '>':
+		ctx.addToken(token.Folded(">"+opt, headerBuf, s.pos()))
+		ctx.isFolded = true
+	}
+	if commentIndex > 0 {
+		comment := string(value[commentValueIndex+1:])
+		s.offset += len(headerBuf)
+		s.column += len(headerBuf)
+		ctx.addToken(token.Comment(comment, string(ctx.obuf[len(headerBuf):]), s.pos()))
+	}
+	s.indentState = IndentStateKeep
+	ctx.resetBuffer()
+	ctx.docOpt = opt
+	s.progressColumn(ctx, progress)
+	return nil
 }
 
 func (s *Scanner) scanMapKey(ctx *Context) bool {
@@ -1063,7 +1189,9 @@ func (s *Scanner) scanDirective(ctx *Context) bool {
 		return false
 	}
 
-	ctx.addToken(token.Directive(string(ctx.obuf)+"%", s.pos()))
+	s.addBufferedTokenIfExists(ctx)
+	ctx.addOriginBuf('%')
+	ctx.addToken(token.Directive(string(ctx.obuf), s.pos()))
 	s.progressColumn(ctx, 1)
 	ctx.clear()
 	s.isDirective = true
@@ -1279,6 +1407,13 @@ func (s *Scanner) scan(ctx *Context) error {
 				return err
 			}
 		case '\t':
+			if ctx.existsBuffer() && s.lastDelimColumn == 0 {
+				// tab indent for plain text (yaml-test-suite's spec-example-7-12-plain-lines).
+				s.indentNum++
+				ctx.addOriginBuf(c)
+				s.progressColumn(ctx, 1)
+				continue
+			}
 			if err := s.scanTab(ctx, c); err != nil {
 				return err
 			}
diff --git a/token/token.go b/token/token.go
index 208ebff..b880bd2 100644
--- a/token/token.go
+++ b/token/token.go
@@ -4,6 +4,7 @@ import (
 	"fmt"
 	"strconv"
 	"strings"
+	"time"
 )
 
 // Character type for character
@@ -217,7 +218,7 @@ func (c CharacterType) String() string {
 	case CharacterTypeIndicator:
 		return "Indicator"
 	case CharacterTypeWhiteSpace:
-		return "WhiteSpcae"
+		return "WhiteSpace"
 	case CharacterTypeMiscellaneous:
 		return "Miscellaneous"
 	case CharacterTypeEscaped:
@@ -612,20 +613,26 @@ func ToNumber(value string) *NumberValue {
 	}
 }
 
-func looksLikeTimeValue(value string) bool {
-	for i, c := range value {
-		switch c {
-		case ':', '1', '2', '3', '4', '5', '6', '7', '8', '9':
-			continue
-		case '0':
-			if i == 0 {
-				return false
-			}
-			continue
+// This is a subset of the formats permitted by the regular expression
+// defined at http://yaml.org/type/timestamp.html. Note that time.Parse
+// cannot handle: "2001-12-14 21:59:43.10 -5" from the examples.
+var timestampFormats = []string{
+	time.RFC3339Nano,
+	"2006-01-02t15:04:05.999999999Z07:00", // RFC3339Nano with lower-case "t".
+	time.DateTime,
+	time.DateOnly,
+
+	// Not in examples, but to preserve backward compatibility by quoting time values.
+	"15:4",
+}
+
+func isTimestamp(value string) bool {
+	for _, format := range timestampFormats {
+		if _, err := time.Parse(format, value); err == nil {
+			return true
 		}
-		return false
 	}
-	return true
+	return false
 }
 
 // IsNeedQuoted whether need quote for passed string or not
@@ -649,7 +656,7 @@ func IsNeedQuoted(value string) bool {
 	case ':', ' ':
 		return true
 	}
-	if looksLikeTimeValue(value) {
+	if isTimestamp(value) {
 		return true
 	}
 	for i, c := range value {


Test patch:
diff --git a/decode_test.go b/decode_test.go
index 9be730e..7d64be5 100644
--- a/decode_test.go
+++ b/decode_test.go
@@ -810,7 +810,7 @@ func TestDecoder(t *testing.T) {
 		},
 		{
 			"a: \"\\0\"\n",
-			map[string]string{"a": "\\0"},
+			map[string]string{"a": "\x00"},
 		},
 		{
 			"b: 2\na: 1\nd: 4\nc: 3\nsub:\n  e: 5\n",
@@ -1107,6 +1107,28 @@ c:
 			"v: あいうえお\nv2: かきくけこ",
 			map[string]string{"v": "あいうえお", "v2": "かきくけこ"},
 		},
+		{
+			`
+- "Fun with \\"
+- "\" \a \b \e \f"
+- "\n \r \t \v \0"
+- "\  \_ \N \L \P \
+  \x41 \u0041 \U00000041"
+`,
+			[]string{"Fun with \\", "\" \u0007 \b \u001b \f", "\n \r \t \u000b \u0000", "\u0020 \u00a0 \u0085 \u2028 \u2029 A A A"},
+		},
+		{
+			`"\ud83e\udd23"`,
+			"🤣",
+		},
+		{
+			`"\uD83D\uDE00\uD83D\uDE01"`,
+			"😀😁",
+		},
+		{
+			`"\uD83D\uDE00a\uD83D\uDE01"`,
+			"😀a😁",
+		},
 	}
 	for _, test := range tests {
 		t.Run(test.source, func(t *testing.T) {
@@ -2076,13 +2098,13 @@ type unmarshalContext struct {
 func (c *unmarshalContext) UnmarshalYAML(ctx context.Context, b []byte) error {
 	v, ok := ctx.Value("k").(int)
 	if !ok {
-		return fmt.Errorf("cannot get valid context")
+		return errors.New("cannot get valid context")
 	}
 	if v != 1 {
-		return fmt.Errorf("cannot get valid context")
+		return errors.New("cannot get valid context")
 	}
 	if string(b) != "1" {
-		return fmt.Errorf("cannot get valid bytes")
+		return errors.New("cannot get valid bytes")
 	}
 	c.v = v
 	return nil
diff --git a/encode_test.go b/encode_test.go
index 2917c85..a8a3205 100644
--- a/encode_test.go
+++ b/encode_test.go
@@ -3,6 +3,7 @@ package yaml_test
 import (
 	"bytes"
 	"context"
+	"errors"
 	"fmt"
 	"math"
 	"reflect"
@@ -273,7 +274,7 @@ func TestEncoder(t *testing.T) {
 			nil,
 		},
 		{
-			"t2: 2018-01-09T10:40:47Z\nt4: 2098-01-09T10:40:47Z\n",
+			"t2: \"2018-01-09T10:40:47Z\"\nt4: \"2098-01-09T10:40:47Z\"\n",
 			map[string]string{
 				"t2": "2018-01-09T10:40:47Z",
 				"t4": "2098-01-09T10:40:47Z",
@@ -717,14 +718,16 @@ func TestEncoder(t *testing.T) {
 		},
 	}
 	for _, test := range tests {
-		var buf bytes.Buffer
-		enc := yaml.NewEncoder(&buf, test.options...)
-		if err := enc.Encode(test.value); err != nil {
-			t.Fatalf("%+v", err)
-		}
-		if test.source != buf.String() {
-			t.Fatalf("expect = [%s], actual = [%s]", test.source, buf.String())
-		}
+		t.Run(test.source, func(t *testing.T) {
+			var buf bytes.Buffer
+			enc := yaml.NewEncoder(&buf, test.options...)
+			if err := enc.Encode(test.value); err != nil {
+				t.Fatalf("%+v", err)
+			}
+			if test.source != buf.String() {
+				t.Fatalf("expect = [%s], actual = [%s]", test.source, buf.String())
+			}
+		})
 	}
 }
 
@@ -1398,10 +1401,10 @@ type marshalContext struct{}
 func (c *marshalContext) MarshalYAML(ctx context.Context) ([]byte, error) {
 	v, ok := ctx.Value("k").(int)
 	if !ok {
-		return nil, fmt.Errorf("cannot get valid context")
+		return nil, errors.New("cannot get valid context")
 	}
 	if v != 1 {
-		return nil, fmt.Errorf("cannot get valid context")
+		return nil, errors.New("cannot get valid context")
 	}
 	return []byte("1"), nil
 }
diff --git a/lexer/lexer_test.go b/lexer/lexer_test.go
index 7179517..8e94c2f 100644
--- a/lexer/lexer_test.go
+++ b/lexer/lexer_test.go
@@ -1347,7 +1347,7 @@ t4: 2098-01-09T10:40:47Z
 					Type:          token.DoubleQuoteType,
 					CharacterType: token.CharacterTypeIndicator,
 					Indicator:     token.QuotedScalarIndicator,
-					Value:         "\\0",
+					Value:         "\x00",
 					Origin:        " \"\\0\"",
 				},
 			},
@@ -3233,6 +3233,26 @@ a: |
 	c
 `,
 		},
+		{
+			name: "invalid UTF-16 character",
+			src:  `"\u00"`,
+		},
+		{
+			name: "invalid UTF-16 surrogate pair length",
+			src:  `"\ud800"`,
+		},
+		{
+			name: "invalid UTF-16 low surrogate prefix",
+			src:  `"\ud800\v"`,
+		},
+		{
+			name: "invalid UTF-16 low surrogate",
+			src:  `"\ud800\u0000"`,
+		},
+		{
+			name: "invalid UTF-32 character",
+			src:  `"\U0000"`,
+		},
 	}
 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
diff --git a/parser/parser_test.go b/parser/parser_test.go
index 14980b5..c82adb7 100644
--- a/parser/parser_test.go
+++ b/parser/parser_test.go
@@ -70,6 +70,7 @@ func TestParser(t *testing.T) {
 		"%YAML 1.2\n---\n",
 		"a: !!binary gIGC\n",
 		"a: !!binary |\n  " + strings.Repeat("kJCQ", 17) + "kJ\n  CQ\n",
+		"v: !!foo 1",
 		"v:\n- A\n- |-\n  B\n  C\n",
 		"v:\n- A\n- >-\n  B\n  C\n",
 		"v: |-\n  0\n",
@@ -1029,14 +1030,6 @@ func TestSyntaxError(t *testing.T) {
 		source string
 		expect string
 	}{
-		{
-			`v: !!int64 2`,
-			`
-[1:4] unknown secondary tag name "!!int64" specified
->  1 | v: !!int64 2
-          ^
-`,
-		},
 		{
 			`
 a:
@@ -1471,6 +1464,14 @@ foo: | # comment
 			yaml: `
 foo: > # comment
   x: 42
+`,
+		},
+		{
+			name: "unattached comment",
+			yaml: `
+# This comment is in its own document
+---
+a: b
 `,
 		},
 	}
diff --git a/token/token_test.go b/token/token_test.go
index 27565fb..e5e5ba8 100644
--- a/token/token_test.go
+++ b/token/token_test.go
@@ -82,6 +82,10 @@ func TestIsNeedQuoted(t *testing.T) {
 		"true",
 		"1.234",
 		"1:1",
+		"2001-12-15T02:59:43.1Z",
+		"2001-12-14t21:59:43.10-05:00",
+		"2001-12-15 2:59:43.10",
+		"2002-12-14",
 		"hoge # comment",
 		"\\0",
 		"#a b",
@@ -128,15 +132,18 @@ func TestIsNeedQuoted(t *testing.T) {
 	}
 	for i, test := range needQuotedTests {
 		if !token.IsNeedQuoted(test) {
-			t.Fatalf("%d: failed to quoted judge for %s", i, test)
+			t.Errorf("%d: failed to quoted judge for %s", i, test)
 		}
 	}
 	notNeedQuotedTests := []string{
 		"Hello World",
+		// time.Parse cannot handle: "2001-12-14 21:59:43.10 -5" from the examples.
+		// https://yaml.org/type/timestamp.html
+		"2001-12-14 21:59:43.10 -5",
 	}
 	for i, test := range notNeedQuotedTests {
 		if token.IsNeedQuoted(test) {
-			t.Fatalf("%d: failed to quoted judge for %s", i, test)
+			t.Errorf("%d: failed to quoted judge for %s", i, test)
 		}
 	}
 }
diff --git a/yaml_test_suite_test.go b/yaml_test_suite_test.go
index 5a54112..346ecc9 100644
--- a/yaml_test_suite_test.go
+++ b/yaml_test_suite_test.go
@@ -14,126 +14,75 @@ import (
 )
 
 var failureTestNames = []string{
-	"anchors-on-empty-scalars",
-	"aliases-in-flow-objects",
-	"aliases-in-explicit-block-mapping",
-	"aliases-in-implicit-block-mapping",
-	"bare-document-after-document-end-marker",
-	"block-mapping-with-missing-keys",
-	"block-mapping-with-missing-values",
-	"block-mapping-with-multiline-scalars",
-	"block-scalar-with-more-spaces-than-first-content-line",
-	"block-scalar-with-wrong-indented-line-after-spaces-only",
+	"anchors-on-empty-scalars",          // no json.
+	"aliases-in-flow-objects",           // no json.
+	"aliases-in-explicit-block-mapping", // no json.
+	"block-mapping-with-missing-keys",   // no json.
 	"colon-at-the-beginning-of-adjacent-flow-scalar",
-	"comment-in-flow-sequence-before-comma",
 	"comment-without-whitespace-after-doublequoted-scalar",
 	"construct-binary",
 	"dash-in-flow-sequence",
-	"directive-variants/00",
-	"directive-variants/01",
-	"double-quoted-scalar-with-escaped-single-quote",
-	"duplicate-yaml-directive",
-	"escaped-slash-in-double-quotes",
-	"explicit-key-and-value-seperated-by-comment", //nolint: misspell
-	"extra-words-on-yaml-directive",
-	"empty-implicit-key-in-single-pair-flow-sequences",
-	"empty-keys-in-block-and-flow-mapping",
-	"empty-lines-at-end-of-document",
-	"flow-mapping-separate-values",
+	"empty-implicit-key-in-single-pair-flow-sequences", // no json.
+	"empty-keys-in-block-and-flow-mapping",             // no json.
+	"empty-lines-at-end-of-document",                   // no json.
+	"flow-mapping-separate-values",                     // no json.
 	"flow-sequence-in-flow-mapping",
 	"flow-collections-over-many-lines/01",
 	"flow-mapping-colon-on-line-after-key/02",
 	"flow-mapping-edge-cases",
-	"flow-sequence-with-invalid-comma-at-the-beginning",
-	"folded-block-scalar",
-	"folded-block-scalar-1-3",
-	"implicit-flow-mapping-key-on-one-line",
-	"invalid-anchor-in-zero-indented-sequence",
+	"flow-sequence-with-invalid-comma-at-the-beginning", // pass yamlv3.
+	"folded-block-scalar",                               // pass yamlv3.
+	"folded-block-scalar-1-3",                           // pass yamlv3.
+	"implicit-flow-mapping-key-on-one-line",             // no json.
 	"invalid-comment-after-comma",
 	"invalid-comment-after-end-of-flow-sequence",
-	"invalid-document-end-marker-in-single-quoted-string",
-	"invalid-document-start-marker-in-doublequoted-tring",
-	"invalid-escape-in-double-quoted-string",
-	"invalid-tag",
-	"key-with-anchor-after-missing-explicit-mapping-value",
-	"leading-tabs-in-double-quoted/02",
-	"leading-tabs-in-double-quoted/05",
-	"legal-tab-after-indentation",
-	"literal-block-scalar-with-more-spaces-in-first-line",
-	"literal-modifers/00",
-	"literal-modifers/01",
-	"literal-modifers/02",
-	"literal-modifers/03",
-	"literal-scalars",
-	"mapping-key-and-flow-sequence-item-anchors",
-	"mapping-starting-at-line",
-	"mapping-with-anchor-on-document-start-line",
-	"multiline-double-quoted-implicit-keys",
-	"multiline-plain-flow-mapping-key",
-	"multiline-plain-value-with-tabs-on-empty-lines",
-	"multiline-scalar-at-top-level",
-	"multiline-scalar-at-top-level-1-3",
-	"multiline-single-quoted-implicit-keys",
-	"multiline-unidented-double-quoted-block-key",
-	"nested-implicit-complex-keys",
-	"node-anchor-not-indented",
+	"invalid-comma-in-tag",
+	"invalid-tag",                                    // pass yamlv3.
+	"legal-tab-after-indentation",                    // pass yamlv3.
+	"literal-scalars",                                // pass yamlv3.
+	"mapping-key-and-flow-sequence-item-anchors",     // no json.
+	"multiline-plain-value-with-tabs-on-empty-lines", // pass yamlv3.
+	"multiline-scalar-at-top-level",                  // pass yamlv3.
+	"multiline-scalar-at-top-level-1-3",              // pass yamlv3.
+	"nested-implicit-complex-keys",                   // no json.
 	"plain-dashes-in-flow-sequence",
-	"plain-url-in-flow-mapping",
-	"question-mark-edge-cases/00",
-	"question-mark-edge-cases/01",
-	"scalar-doc-with-in-content/01",
-	"scalar-value-with-two-anchors",
-	"single-character-streams/01",
-	"single-pair-implicit-entries",
-	"spec-example-2-11-mapping-between-sequences",
-	"spec-example-6-12-separation-spaces",
-	"spec-example-7-16-flow-mapping-entries",
-	"spec-example-7-3-completely-empty-flow-nodes",
-	"spec-example-8-18-implicit-block-mapping-entries",
-	"spec-example-8-19-compact-block-mappings",
-	"spec-example-6-19-secondary-tag-handle",
-	"spec-example-6-21-local-tag-prefix",
-	"spec-example-6-24-verbatim-tags",
-	"spec-example-6-28-non-specific-tags",
-	"spec-example-6-4-line-prefixes",
-	"spec-example-6-6-line-folding",
-	"spec-example-6-6-line-folding-1-3",
-	"spec-example-6-8-flow-folding",
-	"spec-example-7-12-plain-lines",
-	"spec-example-7-19-single-pair-flow-mappings",
-	"spec-example-7-20-single-pair-explicit-entry",
-	"spec-example-7-24-flow-nodes",
-	"spec-example-8-10-folded-lines-8-13-final-empty-lines",
-	"spec-example-8-15-block-sequence-entry-types",
-	"spec-example-8-17-explicit-block-mapping-entries",
+	"question-mark-edge-cases/00",                           // no json.
+	"question-mark-edge-cases/01",                           // no json.
+	"single-character-streams/01",                           // no json.
+	"single-pair-implicit-entries",                          // no json.
+	"spec-example-2-11-mapping-between-sequences",           // no json.
+	"spec-example-6-12-separation-spaces",                   // no json.
+	"spec-example-7-16-flow-mapping-entries",                // no json.
+	"spec-example-7-3-completely-empty-flow-nodes",          // no json.
+	"spec-example-8-18-implicit-block-mapping-entries",      // no json.
+	"spec-example-8-19-compact-block-mappings",              // no json.
+	"spec-example-6-6-line-folding",                         // pass yamlv3.
+	"spec-example-6-6-line-folding-1-3",                     // pass yamlv3.
+	"spec-example-8-10-folded-lines-8-13-final-empty-lines", // pass yamlv3.
 	"spec-example-8-2-block-indentation-indicator",
 	"spec-example-9-3-bare-documents",
 	"spec-example-9-4-explicit-documents",
-	"spec-example-9-5-directives-documents",
 	"spec-example-9-6-stream",
 	"spec-example-9-6-stream-1-3",
-	"syntax-character-edge-cases/00",
+	"syntax-character-edge-cases/00", // no json.
 	"tab-at-beginning-of-line-followed-by-a-flow-mapping",
 	"tab-indented-top-flow",
 	"tabs-in-various-contexts/003",
 	"tabs-that-look-like-indentation/00",
 	"tabs-that-look-like-indentation/01",
-	"tabs-that-look-like-indentation/02",
 	"tabs-that-look-like-indentation/03",
 	"tabs-that-look-like-indentation/04",
-	"tabs-that-look-like-indentation/05",
+	"tabs-that-look-like-indentation/05", // pass yamlv3.
 	"tabs-that-look-like-indentation/07",
-	"tabs-that-look-like-indentation/08",
 	"tag-shorthand-used-in-documents-but-only-defined-in-the-first",
-	"tags-for-block-objects",
-	"tags-on-empty-scalars",
+	"tags-on-empty-scalars",                            // no json.
 	"trailing-line-of-spaces/01",                       // last '\n' character is needed ?
-	"various-combinations-of-explicit-block-mappings",  // no json
-	"various-trailing-comments",                        // no json
-	"various-trailing-comments-1-3",                    // no json
+	"various-combinations-of-explicit-block-mappings",  // no json.
+	"various-trailing-comments",                        // no json.
+	"various-trailing-comments-1-3",                    // no json.
 	"wrong-indented-flow-sequence",                     // error ?
 	"wrong-indented-multiline-quoted-scalar",           // error ?
-	"zero-indented-sequences-in-explicit-mapping-keys", // no json
+	"zero-indented-sequences-in-explicit-mapping-keys", // no json.
 }
 
 var failureTestNameMap map[string]struct{}