Change:
Attempting to create three namespaces (foo, bar, baz). Notice the first YAML document separator has the comment after it on the same line. Output: Exit code is 0 and only and were created. It should either create all three namespaces or else it should fail with an error telling me my YAML is invalid. See Above. Same thing happens if I put random characters after the separator. For example: No error message is printed, exit code is 0, and the next resource is just silently ignored Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
/assign
/sig api-machinery likely something that does not tolerate? you can try reproducing that with a unit test.
Thanks for the hint! I actually had a look at it and the Yaml document's termination logic seems to happen here: We currently assume that three dashes only terminate the document if followed by spaces (or by new line immediately), otherwise we keep scanning the document consider it as a single doc (including three dashes and any characters that follow). Ref: Doing that will make the yaml unmarshalling ignore everything that is followed by these three dashes (as it does not support unmarshalling multiple docs) To prove it: Will output: According to the Yaml Spec it's OK to have any character after these three dashes, but it will be considered part of the new doc already (as three dashes indicate seperation of Yaml directives from document's content) see One thing to note here is we ignore the use of three dots ("...") as a separator of one document's from another's Yaml directives. So for example if we had this kind of Yaml we would consider to be part of the first doc, which is wrong assumption according to the spec. To keep it simple though, I will sign a PR (including tests) later that will handle the three dashes and the followed characters to be a termination indicator and possible taken under consideration for the following (second) doc.
Thanks for having a look.
Let us know when you send a PR, thank you! /triage accepted
I would love to have your opinions on the following: There are 3 fix stages to this bug as I see it: 1) We should split to different Yaml documents when three dashes appears, whether they followed by comments (or any other content), and not only spaces, so we would handle the strange behavior described in the bug description. 2) We should consider three dashes as part of the next document (and not only as a terminator - so for example in we won't be dropping the existence of This one is breaking because when three dashes appear at the end of the document i.e we would consider this file as containing 2 docs (the second is empty). 3) We need to take 3 dots under consideration as described above. This would also be breaking change obviously (and in many more ways in contrast to the 2nd change) and in my opinion needs to be filed as an entirely different issue Each one of these changes get us closer to an alignment state with the Yaml spec. What do you suggest to include for this PR, stage1, or stage 2 as well?
I'm in the process of . To keep it simple, the approach I settled on, at least so far, is to allow (and ignore) white space or comments, but any other content will result in an error. As you pointed out there is a lot more to the YAML syntax. However, it can be quite a bit more work trying to support all the possible things that are syntactically correct but not something that would be used in kubernetes, like . I also chose not to address because my impression was that this is mostly used for "streamed" YAML where there is no EOF. I actually didn't even know that existed before digging into the spec. I'm not saying it would be bad to support that, just that I don't think anyone uses it for their manifests so the "ROI" for making that change would be low.
my vote would be to not introduce breaking changes, because this code path is used by literally everyone with manifest files. note that the same code is used for JSON multi-doc splitting too. i think ideally we should not be splitting at any occurrence of . is the specification explicit about the being at position 0 of a new line? if so the accurate way would be: split at add back a to item add everything after to item (excluding the delimiter ) but that might be breaking already. i think the current behavior is to drop empty docs and the start and end. applies to: resulting in only one doc. this is quite common in manifests. probably we shouldn't manage 3 dots because i haven't seen requests for it. also depends on what the spec says and if the split should happen at and not any . special casing only white space and comments after a makes sense for the time being, IMO.
I agree, breaking changes are dangerous here and should be done gradually (although you right we already apply empty docs as in this case: so taking things like as described above would not be actually breaking. Anyways, I think we'll act cautiously as a result of this change's sensitivity, and go with approach to this here by special casing whitespaces and comments. Thanks for the involvement.
PR is ready. ^

Discussion:
if i := bytes.Index(line, []byte(separator)); i == 0 { // We have a potential document terminator i += sep <del> after := line[i:] if len(strings.TrimRightFunc(string(after), unicode.IsSpace)) == 0 { if buffer.Len() != 0 { return buffer.Bytes(), nil } if err == io.EOF { return nil, err </del> <ins> trimmed := strings.TrimSpace(string(line[i:])) // We only allow comments and spaces following the yaml doc separator, otherwise we'll return an error if len(trimmed) > 0 && string(trimmed[0]) != "#" { return nil, YAMLSyntaxError{ err: fmt.Errorf("invalid Yaml document separator: %s", trimmed), </ins> } } <ins> if buffer.Len() != 0 { return buffer.Bytes(), nil } if err == io.EOF { return nil, err } </ins> } if err == io.EOF { if buffer.Len() != 0 {

} } <ins> func TestDecodeYAMLSeparatorValidation(t *testing.T) { s := NewYAMLToJSONDecoder(bytes.NewReader([]byte(`--- stuff: 1 ---    # Make sure termination happen with inline comment stuff: 2 --- stuff: 3 --- Make sure uncommented content results YAMLSyntaxError `))) obj := generic{} if err := s.Decode(&obj); err != nil { t.Fatalf("unexpected error: %v", err) } if fmt.Sprintf("%#v", obj) != `yaml.generic{"stuff":1}` { t.Errorf("unexpected object: %#v", obj) } obj = generic{} if err := s.Decode(&obj); err != nil { t.Fatalf("unexpected error: %v", err) } if fmt.Sprintf("%#v", obj) != `yaml.generic{"stuff":2}` { t.Errorf("unexpected object: %#v", obj) } obj = generic{} err := s.Decode(&obj) if err == nil { t.Fatalf("expected YamlSyntaxError, got nil instead") } if _, ok := err.(YAMLSyntaxError); !ok { t.Fatalf("unexpected error: %v", err) } } </ins> func TestDecodeBrokenYAML(t *testing.T) { s := NewYAMLOrJSONDecoder(bytes.NewReader([]byte(`--- stuff: 1

{"foo": "bar"}, {"baz": "biz"}, }}, <ins> {"---nfoo: barn--- # with Commentnbaz: biz", 100, false, false, []generic{ {"foo": "bar"}, {"baz": "biz"}, }}, </ins> {"foo: barn---n", 100, false, false, []generic{ {"foo": "bar"}, }},