nebius/SWE-rebench-V2-PRs huggingface dataset:
Repository: jhillyerd/enmime
Pull request #220 (2021-09-29 10:39:37)

Fixed crash in QPCleaner, when line is too long and buffer is almost full
Hi,

there is a bug in new QPCleaner implementation, which causes panic. When the `qp.lineLen` exceeds `MaxQPLineLen`, newline is safely inserted to `dest` buffer. But it is not checked, if `n` is still `< destLen` after this insertion, causing write after the end of `dest` in the following `switch` statement.

Here is a fix and test, the test causes panic on current version, the checks are there just for completeness.

Best regards

Pavel

jhillyerd: Thanks for catching this!

Patch:
diff --git a/internal/coding/quotedprint.go b/internal/coding/quotedprint.go
index da37357..6610a7a 100644
--- a/internal/coding/quotedprint.go
+++ b/internal/coding/quotedprint.go
@@ -84,6 +84,9 @@ func (qp *QPCleaner) Read(dest []byte) (n int, err error) {
 		if qp.lineLen >= MaxQPLineLen {
 			writeBytes(lineBreak)
 			qp.lineLen = 0
+			if n == destLen {
+				break
+			}
 		}
 
 		switch {


Test patch:
diff --git a/internal/coding/quotedprint_test.go b/internal/coding/quotedprint_test.go
index ed2f091..2d3ea0a 100644
--- a/internal/coding/quotedprint_test.go
+++ b/internal/coding/quotedprint_test.go
@@ -137,6 +137,21 @@ func TestQPCleanerLineBreak(t *testing.T) {
 	}
 }
 
+func TestQPCleanerLineBreakBufferFull(t *testing.T) {
+	input := bytes.Repeat([]byte("abc"), 10000)
+	inbuf := bytes.NewBuffer(input)
+	qp := coding.NewQPCleaner(inbuf)
+
+	dest := make([]byte, 1025)
+	n, err := qp.Read(dest)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if n != 1025 {
+		t.Errorf("Unexpected result length: %d", n)
+	}
+}
+
 var ErrPeek = errors.New("enmime test peek error")
 
 type peekBreakReader string
Repository: jhillyerd/enmime
Pull request #224 (2022-01-03 14:50:44)

Fix newlines in mediatype.Parse
Added a very simple `fixNewlines` function to handle the following outlook produced case: 

```
		{
			label:  "Outlook-Logo with newlines",
			input:  `application/octet-stream; name="=?utf-8?B?T3V0bG9vay1Mb2dvCgpEZXNj?="`,
			mtype:  "application/octet-stream",
			params: map[string]string{"name": "Outlook-Logo  Desc"},
		},
```

jhillyerd: While I agree that I wouldn't want a newline in a filename, I'm not sure this is something that should be fixed in enmime, since clients can choose how to handle newlines themselves?  I've tried to add well meaning things like this to enmime in the past and caused problems for others.
ungerik: I wouldn't mind the newlines, but the go std package refuses to parse values with newlines.
jhillyerd: That sounds like a good reason to me, thanks.

Patch:
diff --git a/mediatype/mediatype.go b/mediatype/mediatype.go
index 7c3a026..e492261 100644
--- a/mediatype/mediatype.go
+++ b/mediatype/mediatype.go
@@ -39,9 +39,10 @@ const (
 //   * Missing ';' between content-type and media parameters
 //   * Repeating media parameters
 //   * Unquoted values in media parameters containing 'tspecials' characters
+//   * Newline characters
 func Parse(ctype string) (mtype string, params map[string]string, invalidParams []string, err error) {
 	mtype, params, err = mime.ParseMediaType(
-		fixUnescapedQuotes(fixUnquotedSpecials(fixMangledMediaType(ctype, ';'))))
+		fixNewlines(fixUnescapedQuotes(fixUnquotedSpecials(fixMangledMediaType(ctype, ';')))))
 	if err != nil {
 		if err.Error() == "mime: no media type" {
 			return "", nil, nil, nil
@@ -471,3 +472,10 @@ func fixUnescapedQuotes(hvalue string) string {
 
 	return sb.String()
 }
+
+// fixNewlines replaces \n with a space and removes \r
+func fixNewlines(value string) string {
+	value = strings.ReplaceAll(value, "\n", " ")
+	value = strings.ReplaceAll(value, "\r", "")
+	return value
+}


Test patch:
diff --git a/mediatype/mediatype_test.go b/mediatype/mediatype_test.go
index 2df7914..228a38d 100644
--- a/mediatype/mediatype_test.go
+++ b/mediatype/mediatype_test.go
@@ -458,6 +458,12 @@ func TestParseMediaType(t *testing.T) {
 			mtype:  "application/pdf",
 			params: map[string]string{"x-unix-mode": "0644", "name": "File name with spaces.pdf"},
 		},
+		{
+			label:  "Outlook-Logo with newlines",
+			input:  `application/octet-stream; name="=?utf-8?B?T3V0bG9vay1Mb2dvCgpEZXNj?="`,
+			mtype:  "application/octet-stream",
+			params: map[string]string{"name": "Outlook-Logo  Desc"},
+		},
 	}
 	for _, tc := range testCases {
 		t.Run(tc.label, func(t *testing.T) {
Repository: jhillyerd/enmime
Pull request #225 (2022-01-17 11:19:47)

fix(fixMangledMediaType) remove extra parts
Hello,

I encountered some unknown email client in production. It used an invalid content type with extra parts, like `application/pdf/.pdf`

I `fixed` parsing by removing extra parts, but I'm not 100% sure it could be split as 
* `application` `pdf`
* `application` `pdf/.pdf`
* `application` `pdf-.pdf`

Thank you again for your great work

jhillyerd: I think this is a good start, and if folks run into issues we can try to tune it.

Patch:
diff --git a/mediatype/mediatype.go b/mediatype/mediatype.go
index e492261..85a87cc 100644
--- a/mediatype/mediatype.go
+++ b/mediatype/mediatype.go
@@ -101,6 +101,11 @@ func fixMangledMediaType(mtype string, sep rune) string {
 					p = ctAppOctetStream
 				}
 			}
+			// Remove extra ctype parts
+			if strings.Count(p, "/") > 1 {
+				ps := strings.SplitN(p, "/", 3)
+				p = strings.Join(ps[0:2], "/")
+			}
 		default:
 			if len(p) == 0 {
 				// Ignore trailing separators.


Test patch:
diff --git a/mediatype/mediatype_test.go b/mediatype/mediatype_test.go
index 228a38d..37e252c 100644
--- a/mediatype/mediatype_test.go
+++ b/mediatype/mediatype_test.go
@@ -115,6 +115,18 @@ func TestFixMangledMediaType(t *testing.T) {
 			sep:   ';',
 			want:  `one/two; name="file.two"`,
 		},
+
+		// remove extra content type parts
+		{
+			input: `application/pdf/.pdf; name=1337.pdf`,
+			sep:   ';',
+			want:  `application/pdf; name=1337.pdf`,
+		},
+		{
+			input: `application/pdf/pdf/pdf; name=1337.pdf`,
+			sep:   ';',
+			want:  `application/pdf; name=1337.pdf`,
+		},
 	}
 	for _, tc := range testCases {
 		t.Run(tc.input, func(t *testing.T) {
Repository: jhillyerd/enmime
Pull request #253 (2022-05-31 19:22:40)

feat: Add Envelope.Date() method to parse the mail date


Patch:
diff --git a/envelope.go b/envelope.go
index ff7e882..039524c 100644
--- a/envelope.go
+++ b/envelope.go
@@ -7,6 +7,7 @@ import (
 	"net/mail"
 	"net/textproto"
 	"strings"
+	"time"
 
 	"github.com/jaytaylor/html2text"
 	"github.com/jhillyerd/enmime/internal/coding"
@@ -114,6 +115,15 @@ func (e *Envelope) AddressList(key string) ([]*mail.Address, error) {
 	return ParseAddressList(e.header.Get(key))
 }
 
+// Date parses the Date header field.
+func (e *Envelope) Date() (time.Time, error) {
+	hdr := e.GetHeader("Date")
+	if hdr == "" {
+		return time.Time{}, mail.ErrHeaderNotPresent
+	}
+	return mail.ParseDate(hdr)
+}
+
 // Clone returns a clone of the current Envelope
 func (e *Envelope) Clone() *Envelope {
 	if e == nil {


Test patch:
diff --git a/envelope_test.go b/envelope_test.go
index 8ad5185..c1f47ef 100644
--- a/envelope_test.go
+++ b/envelope_test.go
@@ -7,6 +7,7 @@ import (
 	"sort"
 	"strings"
 	"testing"
+	"time"
 
 	"github.com/go-test/deep"
 	"github.com/jhillyerd/enmime"
@@ -34,6 +35,12 @@ func TestParseHeaderOnly(t *testing.T) {
 	if len(e.Root.Header) != 7 {
 		t.Errorf("Expected 7 headers, got %d", len(e.Root.Header))
 	}
+	dt, err := e.Date()
+	if err != nil {
+		t.Errorf("Failed to parse Date header: %v", err)
+	} else if !dt.Equal(time.Date(2012, time.October, 19, 5, 48, 39, 0, time.UTC)) {
+		t.Errorf("Incorrect date parsed: %s", dt)
+	}
 }
 
 func TestParseNonMime(t *testing.T) {
Repository: jhillyerd/enmime
Pull request #255 (2022-06-07 06:40:33)

fix: QPCleaner line length equal sign counting
Hi,

as mentioned in https://github.com/jhillyerd/enmime/pull/254, this fixes missing counting of `=` when quoted printable encoded char is processed.

I've also mentioned, that line length is not incremented, when `qp.overflow` is put into the buffer at the beginning of the next `Read`. Tests show it is not needed, because `qp.lineLength` is set, when `qp.overflow` is filled at the end of previous `Read`.

Regards


Patch:
diff --git a/internal/coding/quotedprint.go b/internal/coding/quotedprint.go
index 3e2c81f..6ebeb7d 100644
--- a/internal/coding/quotedprint.go
+++ b/internal/coding/quotedprint.go
@@ -52,7 +52,7 @@ func (qp *QPCleaner) Read(dest []byte) (n int, err error) {
 		qp.lineLen++
 	}
 
-	// safeWriteByte outputs a signle byte, storing overflow for next read. Updates counters.
+	// safeWriteByte outputs a single byte, storing overflow for next read. Updates counters.
 	safeWriteByte := func(in byte) {
 		if n < destLen {
 			dest[n] = in
@@ -60,6 +60,7 @@ func (qp *QPCleaner) Read(dest []byte) (n int, err error) {
 		} else {
 			qp.overflow = append(qp.overflow, in)
 		}
+		qp.lineLen++
 	}
 
 	// writeBytes outputs multiple bytes, storing overflow for next read. Updates counters.


Test patch:
diff --git a/internal/coding/quotedprint_test.go b/internal/coding/quotedprint_test.go
index 36e407e..f6e2b60 100644
--- a/internal/coding/quotedprint_test.go
+++ b/internal/coding/quotedprint_test.go
@@ -205,6 +205,39 @@ func TestQPPeekError(t *testing.T) {
 	}
 }
 
+func TestQPCleanerQuotedLineLength(t *testing.T) {
+	input := strings.Repeat("=BC", 700) // ~ two lines of token
+	inr := strings.NewReader(input)
+	qp := coding.NewQPCleaner(inr)
+
+	// Check line length is counted proerly even for quoted printable encoded chars
+	longLineLen := coding.MaxQPLineLen + 2
+	output := make([]byte, longLineLen)
+	n, err := qp.Read(output)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if n != longLineLen {
+		t.Error("got:", n, "want:", longLineLen)
+	}
+	if string(output[longLineLen-2:]) != "\r\n" {
+		t.Error("got:", string(output[longLineLen-2:]), "want:", "\r\n")
+	}
+
+	// Check line length is correct also when overflow buffer of QPCleaner is used
+	output = make([]byte, longLineLen)
+	n, err = qp.Read(output)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if n != longLineLen {
+		t.Error("got:", n, "want:", longLineLen)
+	}
+	if string(output[longLineLen-2:]) != "\r\n" {
+		t.Error("got:", string(output[longLineLen-2:]), "want:", "\r\n")
+	}
+}
+
 var result int
 
 func BenchmarkQPCleaner(b *testing.B) {
Repository: jhillyerd/enmime
Pull request #278 (2023-02-12 16:18:41)

feat: export DecodeRFC2047 (#277)
Exports coding.RFC2047Decode(), making the large set of charsets more readily available for package users.


Patch:
diff --git a/inspect.go b/inspect.go
index 82b8e13..9652a96 100644
--- a/inspect.go
+++ b/inspect.go
@@ -20,6 +20,13 @@ var defaultHeadersList = []string{
 	"Date",
 }
 
+// DecodeRFC2047 decodes the given string according to RFC 2047 and returns the
+// decoded UTF-8 equivalent. If the input is not using RFC 2047 encoding, or the
+// charset is not recognized, it will return the input unmodified.
+func DecodeRFC2047(s string) string {
+	return coding.RFC2047Decode(s)
+}
+
 // DecodeHeaders returns a limited selection of mime headers for use by user agents
 // Default header list:
 //
@@ -45,7 +52,7 @@ func DecodeHeaders(b []byte, addtlHeaders ...string) (textproto.MIMEHeader, erro
 		h := textproto.CanonicalMIMEHeaderKey(header)
 		res[h] = make([]string, 0, len(headers[h]))
 		for _, value := range headers[h] {
-			res[h] = append(res[h], coding.RFC2047Decode(value))
+			res[h] = append(res[h], DecodeRFC2047(value))
 		}
 	}
 


Test patch:
diff --git a/inspect_test.go b/inspect_test.go
index 826e05f..a85856a 100644
--- a/inspect_test.go
+++ b/inspect_test.go
@@ -13,6 +13,29 @@ import (
 	"github.com/jhillyerd/enmime/internal/test"
 )
 
+func TestDecodeRFC2047(t *testing.T) {
+	t.Run("rfc2047 basic", func(t *testing.T) {
+		s := enmime.DecodeRFC2047("=?UTF-8?Q?Miros=C5=82aw_Marczak?=")
+		if s != "Mirosław Marczak" {
+			t.Errorf("Wrong decoded result")
+		}
+	})
+
+	t.Run("rfc2047 unknown", func(t *testing.T) {
+		s := enmime.DecodeRFC2047("=?ABC-1?Q?FooBar?=")
+		if s != "=?ABC-1?Q?FooBar?=" {
+			t.Errorf("Expected unmodified result for unknown charset")
+		}
+	})
+
+	t.Run("rfc2047 pass-through", func(t *testing.T) {
+		s := enmime.DecodeRFC2047("Hello World")
+		if s != "Hello World" {
+			t.Errorf("Expected unmodified result")
+		}
+	})
+}
+
 func TestDecodeHeaders(t *testing.T) {
 	t.Run("rfc2047 sample", func(t *testing.T) {
 		r := test.OpenTestData("mail", "qp-utf8-header.raw")
Repository: jhillyerd/enmime
Pull request #289 (2023-04-27 13:12:12)

fix for values with encoded equal signs
We faced the issue that emails with encoded equal signs could not be parsed in certain cases. This fix should improve the situation. 
Thank you in advance for any code reviews and great library!

iredmail: Better use `strings.Cut()` instead of `strings.SplitAfterN()`.

- `strings.Cut()` was introduced in Go 1.18.
- enmime requires Go 1.18 too as defined in `go.mod`.

Patch:
diff --git a/internal/coding/headerext.go b/internal/coding/headerext.go
index a463208..8b54512 100644
--- a/internal/coding/headerext.go
+++ b/internal/coding/headerext.go
@@ -54,20 +54,20 @@ func RFC2047Decode(s string) string {
 
 		default:
 			if decoded {
-				keyValuePair := strings.SplitAfter(s, "=")
-				if len(keyValuePair) < 2 {
+				key, value, found := strings.Cut(s, "=")
+				if !found {
 					return s
 				}
 
 				// Add quotes as needed.
-				if !strings.HasPrefix(keyValuePair[1], "\"") {
-					keyValuePair[1] = fmt.Sprintf("\"%s", keyValuePair[1])
+				if !strings.HasPrefix(value, "\"") {
+					value = fmt.Sprintf("\"%s", value)
 				}
-				if !strings.HasSuffix(keyValuePair[1], "\"") {
-					keyValuePair[1] = fmt.Sprintf("%s\"", keyValuePair[1])
+				if !strings.HasSuffix(value, "\"") {
+					value = fmt.Sprintf("%s\"", value)
 				}
 
-				return strings.Join(keyValuePair, "")
+				return fmt.Sprintf("%s=%s", key, value)
 			}
 
 			return s


Test patch:
diff --git a/mediatype/mediatype_test.go b/mediatype/mediatype_test.go
index 94f6062..56509d4 100644
--- a/mediatype/mediatype_test.go
+++ b/mediatype/mediatype_test.go
@@ -497,6 +497,12 @@ func TestParseMediaType(t *testing.T) {
 			mtype:  "multipart/alternative",
 			params: map[string]string{"boundary": "<myboundary>"},
 		},
+		{
+			label:  "encoded equal sign",
+			input:  "application/pdf; name=\"=?iso-8859-1?Q?key=3Dvalue?=\"",
+			mtype:  "application/pdf",
+			params: map[string]string{"name": "key=value"},
+		},
 	}
 	for _, tc := range testCases {
 		t.Run(tc.label, func(t *testing.T) {
Repository: jhillyerd/enmime
Pull request #313 (2023-12-12 11:23:18)

feat: skip empty parameters
Looks like there was never a proper condition for filtering out empty parameters. It kinda worked thanks to this https://github.com/jhillyerd/enmime/blob/main/mediatype/mediatype.go#L125-L128 (`=` substring would always match if there was at least one parameter already added to `mtype`). It did not work if the empty parameter was the first parameter.

coveralls: 
[![Coverage Status](https://coveralls.io/builds/64509368/badge)](https://coveralls.io/builds/64509368)

coverage: 86.213% (+0.01%) from 86.199%
when pulling **4d65648678cec7be30b99acd1c2bb00a032e2413 on milankonir:fix-mangled-unnamed-parameter**
into **ea70bfea9ada861ce060092bb38af189e8fe0a7e on jhillyerd:main**.


Patch:
diff --git a/mediatype/mediatype.go b/mediatype/mediatype.go
index 702d9b7..e9e44c8 100644
--- a/mediatype/mediatype.go
+++ b/mediatype/mediatype.go
@@ -122,6 +122,12 @@ func fixMangledMediaType(mtype string, sep rune) string {
 			p = coding.RFC2047Decode(p)
 
 			pair := strings.SplitAfter(p, "=")
+
+			if strings.TrimSpace(pair[0]) == "=" {
+				// Ignore unnamed parameters.
+				continue
+			}
+
 			if strings.Contains(mtype, strings.TrimSpace(pair[0])) {
 				// Ignore repeated parameters.
 				continue


Test patch:
diff --git a/mediatype/mediatype_test.go b/mediatype/mediatype_test.go
index 56509d4..fa1fd12 100644
--- a/mediatype/mediatype_test.go
+++ b/mediatype/mediatype_test.go
@@ -46,6 +46,12 @@ func TestFixMangledMediaType(t *testing.T) {
 			sep:   ';',
 			want:  `image/png; name="abc.png"`,
 		},
+		{
+			// Removes empty parameters in the middle
+			input: `Content-Type: text/html; =""; charset=""`,
+			sep:   ';',
+			want:  `Content-Type: text/html; charset=""`,
+		},
 		{
 			input: "application/octet-stream;=?UTF-8?B?bmFtZT0iw7DCn8KUwoo=?=You've got a new voice miss call.msg",
 			sep:   ';',