Repository: fxamacker/cbor
Pull request #398 (2023-04-28 17:44:28)

Add UnmarshalFirst
### Description

In a similar vein to #397, there are some places in our code where we have a CBOR header in front of non-CBOR data. As we started integrating the decoder-based approach, we found there were places where we already had the data as a slice (not a reader), and the data items were sufficiently small that the overhead of constructing a reader over the slice so as to use the decoder became significant [1].

As a separate, but related issue, after we upgraded to 2.5.0, we found there were places in our code that relied upon the old behavior of Unmarshal being OK with the data slice having extraneous data, and it took us a little bit to find the issue as the comment on Unmarshal doesn't explicitly mention the new behavior.

This PR hopes to solve both of those issues by:
- Adding an UnmarshalFirst (neé UnmarshalPrefix) function that behaves similarly to the ASN.1 and PEM functions I referenced in #397. Namely, it is exactly like Unmarshal except that it is ok with having extraneous data and will return the remaining data to you.
- Adding a comment to Unmarshal that makes it clear that an ExtraneousDataError is returned if there is data appearing after a valid item. I think this makes it clearer to those observing errors after upgrading to 2.5.0, what the cause is and what they may want to do about it (change data, or use UnmarshalFirst)

[1] This PR also adds two benchmarks that show what I mean about performance difference between Unmarshal and Decode for smaller items. An excerpt of that from my Macbook Pro M1
```

BenchmarkUnmarshalPrefix/CBOR_bool_to_Go_bool-10                 	27587844	        43.44 ns/op
BenchmarkUnmarshalPrefix/CBOR_positive_int_to_Go_uint64-10       	23871630	        50.57 ns/op
BenchmarkUnmarshalPrefix/CBOR_float_to_Go_float64-10             	23281004	        52.00 ns/op
BenchmarkUnmarshalPrefix/CBOR_bytes_to_Go_[]uint8-10             	12625482	        93.86 ns/op
BenchmarkUnmarshalPrefix/CBOR_map_to_Go_map[string]string-10                	 1000000	      1197 ns/op
BenchmarkUnmarshalPrefixViaDecoder/CBOR_bool_to_Go_bool-10                           	 7343330	       168.1 ns/op
BenchmarkUnmarshalPrefixViaDecoder/CBOR_positive_int_to_Go_uint64-10                 	 7209103	       154.5 ns/op
BenchmarkUnmarshalPrefixViaDecoder/CBOR_float_to_Go_float64-10                       	 6824787	       159.2 ns/op
BenchmarkUnmarshalPrefixViaDecoder/CBOR_bytes_to_Go_[]uint8-10                       	 6131350	       211.1 ns/op
BenchmarkUnmarshalPrefixViaDecoder/CBOR_map_to_Go_map[string]string-10               	  908347	      1273 ns/op
```

#### PR Was Proposed and Welcomed in Currently Open Issue

I skipped the "propose in issue" step again, but as before, please feel free to propose a completely different solution and I will change the code (or to reject the idea entirely). No hard feelings.

#### Checklist (for code PR only, ignore for docs PR)

- [X] Include unit tests that cover the new code
- [X] Pass all unit tests 
- [x] Pass all 18 ci linters (golint, gosec, staticcheck, etc.)
- [X] Sign each commit with your real name and email.  
      Last line of each commit message should be in this format:  
      Signed-off-by: Firstname Lastname <firstname.lastname@example.com>
- [X] Certify the Developer's Certificate of Origin 1.1
      (see next section).

#### Certify the Developer's Certificate of Origin 1.1

- [X] By marking this item as completed, I certify 
      the Developer Certificate of Origin 1.1.

```
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
660 York Street, Suite 102,
San Francisco, CA 94110 USA

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.
```



immesys: A few rough spots with lint checks and accidentally using go1.20 features, but I think it's good to review now.

Patch:
diff --git a/decode.go b/decode.go
index 5bc9e86..dbfc069 100644
--- a/decode.go
+++ b/decode.go
@@ -95,10 +95,25 @@ import (
 //
 // Unmarshal supports CBOR tag 55799 (self-describe CBOR), tag 0 and 1 (time),
 // and tag 2 and 3 (bignum).
+//
+// Unmarshal returns ExtraneousDataError error (without decoding into v)
+// if there are any remaining bytes following the first valid CBOR data item.
+// See UnmarshalFirst, if you want to unmarshal only the first
+// CBOR data item without ExtraneousDataError caused by remaining bytes.
 func Unmarshal(data []byte, v interface{}) error {
 	return defaultDecMode.Unmarshal(data, v)
 }
 
+// UnmarshalFirst parses the first CBOR data item into the value pointed to by v
+// using default decoding options.  Any remaining bytes are returned in rest.
+//
+// If v is nil, not a pointer, or a nil pointer, UnmarshalFirst returns an error.
+//
+// See the documentation for Unmarshal for details.
+func UnmarshalFirst(data []byte, v interface{}) (rest []byte, err error) {
+	return defaultDecMode.UnmarshalFirst(data, v)
+}
+
 // Valid checks whether data is a well-formed encoded CBOR data item and
 // that it complies with default restrictions such as MaxNestedLevels,
 // MaxArrayElements, MaxMapPairs, etc.
@@ -604,6 +619,35 @@ func (dm *decMode) Unmarshal(data []byte, v interface{}) error {
 	return d.value(v)
 }
 
+// UnmarshalFirst parses the first CBOR data item into the value pointed to by v
+// using dm decoding mode.  Any remaining bytes are returned in rest.
+//
+// If v is nil, not a pointer, or a nil pointer, UnmarshalFirst returns an error.
+//
+// See the documentation for Unmarshal for details.
+func (dm *decMode) UnmarshalFirst(data []byte, v interface{}) (rest []byte, err error) {
+	d := decoder{data: data, dm: dm}
+
+	// check well-formedness.
+	off := d.off             // Save offset before data validation
+	err = d.wellformed(true) // allow extra data after well-formed data item
+	d.off = off              // Restore offset
+
+	// If it is well-formed, parse the value. This is structured like this to allow
+	// better test coverage
+	if err == nil {
+		err = d.value(v)
+	}
+
+	// If either wellformed or value returned an error, do not return rest bytes
+	if err != nil {
+		return nil, err
+	}
+
+	// Return the rest of the data slice (which might be len 0)
+	return d.data[d.off:], nil
+}
+
 // Valid checks whether data is a well-formed encoded CBOR data item and
 // that it complies with configurable restrictions such as MaxNestedLevels,
 // MaxArrayElements, MaxMapPairs, etc.


Test patch:
diff --git a/bench_test.go b/bench_test.go
index 761bff9..5445db2 100644
--- a/bench_test.go
+++ b/bench_test.go
@@ -211,6 +211,54 @@ func BenchmarkUnmarshal(b *testing.B) {
 	}
 }
 
+func BenchmarkUnmarshalFirst(b *testing.B) {
+	// Random trailing data
+	trailingData := hexDecode("4a6b0f4718c73f391091ea1c")
+	for _, bm := range decodeBenchmarks {
+		for _, t := range bm.decodeToTypes {
+			name := "CBOR " + bm.name + " to Go " + t.String()
+			if t.Kind() == reflect.Struct {
+				name = "CBOR " + bm.name + " to Go " + t.Kind().String()
+			}
+			data := make([]byte, 0, len(bm.cborData)+len(trailingData))
+			data = append(data, bm.cborData...)
+			data = append(data, trailingData...)
+			b.Run(name, func(b *testing.B) {
+				for i := 0; i < b.N; i++ {
+					vPtr := reflect.New(t).Interface()
+					if _, err := UnmarshalFirst(data, vPtr); err != nil {
+						b.Fatal("UnmarshalFirst:", err)
+					}
+				}
+			})
+		}
+	}
+}
+
+func BenchmarkUnmarshalFirstViaDecoder(b *testing.B) {
+	// Random trailing data
+	trailingData := hexDecode("4a6b0f4718c73f391091ea1c")
+	for _, bm := range decodeBenchmarks {
+		for _, t := range bm.decodeToTypes {
+			name := "CBOR " + bm.name + " to Go " + t.String()
+			if t.Kind() == reflect.Struct {
+				name = "CBOR " + bm.name + " to Go " + t.Kind().String()
+			}
+			data := make([]byte, 0, len(bm.cborData)+len(trailingData))
+			data = append(data, bm.cborData...)
+			data = append(data, trailingData...)
+			b.Run(name, func(b *testing.B) {
+				for i := 0; i < b.N; i++ {
+					vPtr := reflect.New(t).Interface()
+					if err := NewDecoder(bytes.NewReader(data)).Decode(vPtr); err != nil {
+						b.Fatal("UnmarshalDecoder:", err)
+					}
+				}
+			})
+		}
+	}
+}
+
 func BenchmarkDecode(b *testing.B) {
 	for _, bm := range decodeBenchmarks {
 		for _, t := range bm.decodeToTypes {
diff --git a/decode_test.go b/decode_test.go
index 0e585d3..cc0e983 100644
--- a/decode_test.go
+++ b/decode_test.go
@@ -5981,3 +5981,60 @@ func TestUnmarshalToDefaultMapType(t *testing.T) {
 		})
 	}
 }
+
+func TestUnmarshalFirstNoTrailing(t *testing.T) {
+	for _, tc := range unmarshalTests {
+		var v interface{}
+		if rest, err := UnmarshalFirst(tc.cborData, &v); err != nil {
+			t.Errorf("UnmarshalFirst(0x%x) returned error %v", tc.cborData, err)
+		} else {
+			if len(rest) != 0 {
+				t.Errorf("UnmarshalFirst(0x%x) returned rest %x (want [])", tc.cborData, rest)
+			}
+			// Check the value as well, although this is covered by other tests
+			if tm, ok := tc.emptyInterfaceValue.(time.Time); ok {
+				if vt, ok := v.(time.Time); !ok || !tm.Equal(vt) {
+					t.Errorf("UnmarshalFirst(0x%x) = %v (%T), want %v (%T)", tc.cborData, v, v, tc.emptyInterfaceValue, tc.emptyInterfaceValue)
+				}
+			} else if !reflect.DeepEqual(v, tc.emptyInterfaceValue) {
+				t.Errorf("UnmarshalFirst(0x%x) = %v (%T), want %v (%T)", tc.cborData, v, v, tc.emptyInterfaceValue, tc.emptyInterfaceValue)
+			}
+		}
+	}
+}
+
+func TestUnmarshalfirstTrailing(t *testing.T) {
+	// Random trailing data
+	trailingData := hexDecode("4a6b0f4718c73f391091ea1c")
+	for _, tc := range unmarshalTests {
+		data := make([]byte, 0, len(tc.cborData)+len(trailingData))
+		data = append(data, tc.cborData...)
+		data = append(data, trailingData...)
+		var v interface{}
+		if rest, err := UnmarshalFirst(data, &v); err != nil {
+			t.Errorf("UnmarshalFirst(0x%x) returned error %v", data, err)
+		} else {
+			if !bytes.Equal(trailingData, rest) {
+				t.Errorf("UnmarshalFirst(0x%x) returned rest %x (want %x)", data, rest, trailingData)
+			}
+			// Check the value as well, although this is covered by other tests
+			if tm, ok := tc.emptyInterfaceValue.(time.Time); ok {
+				if vt, ok := v.(time.Time); !ok || !tm.Equal(vt) {
+					t.Errorf("UnmarshalFirst(0x%x) = %v (%T), want %v (%T)", data, v, v, tc.emptyInterfaceValue, tc.emptyInterfaceValue)
+				}
+			} else if !reflect.DeepEqual(v, tc.emptyInterfaceValue) {
+				t.Errorf("UnmarshalFirst(0x%x) = %v (%T), want %v (%T)", data, v, v, tc.emptyInterfaceValue, tc.emptyInterfaceValue)
+			}
+		}
+	}
+}
+
+func TestUnmarshalFirstInvalidItem(t *testing.T) {
+	// UnmarshalFirst should not return "rest" if the item was not well-formed
+	invalidCBOR := hexDecode("83FF20030102")
+	var v interface{}
+	rest, err := UnmarshalFirst(invalidCBOR, &v)
+	if rest != nil {
+		t.Errorf("UnmarshalFirst(0x%x) = (%x, %v), want (nil, err)", invalidCBOR, rest, err)
+	}
+}