whiteout-io/imap-handler — pull request #8 by asutherland (2014-11-11 22:13:46+00:00)
Title: Trailing whitespace in LISTs should not be fatal

On an account hosted at yahoo.co.jp one of the messages you get at signup has a
BODYSTRUCTURE that includes trailing whitespace in the nested paren list rep
for multipart messages, specifically, whitespace after the "body-fld-dsp" and
preceding "body-fld-lang" per the grammar.

The whitespace violates RFC 3501 but it seems silly to fail to parse the
BODYSTRUCTURE in this case.

The raw FETCH line is:
```
* 2 FETCH (BODYSTRUCTURE (("text" "plain" ("charset" "ISO-2022-JP") NIL NIL "7bit" 2515 64 NIL NIL NIL NIL)("text" "html" ("charset" "ISO-2022-JP") NIL NIL "7bit" 7723 151 NIL NIL NIL NIL) "alternative" ("boundary" "pzu8t3naw7b5therma1t") NIL ) FLAGS (\Seen) INTERNALDATE "30-Oct-2014 03:23:45 +0000" UID 2 BODY[HEADER.FIELDS (FROM TO CC BCC SUBJECT REPLY-TO MESSAGE-ID REFERENCES)] {312}
Message-Id: <5451af3d-000e6070-af32c615041a5c3dae408c7ef67e5444-18793@tx203.storage.kks.yahoo.co.jp>
From: box-master@mail.yahoo.co.jp
To: gaialibs@yahoo.co.jp
Subject: Yahoo! JAPAN =?ISO-2022-JP?B?SUQbJEIkNEVQTz8kTiQqNVJNTSRYIUobKEJZYWhvbyE=?=
=?ISO-2022-JP?B?GyRCJVwlQyUvJTkzK0BfJEskRCQkJEYhSxsoQg==?=

)
```

The relevant excerpts of the MIME message are:
```
Message-Id: <5451af3d-000e6070-af32c615041a5c3dae408c7ef67e5444-18793@tx203.storage.kks.yahoo.co.jp>
From: box-master@mail.yahoo.co.jp
To: gaialibs@yahoo.co.jp
Subject: Yahoo! JAPAN =?ISO-2022-JP?B?SUQbJEIkNEVQTz8kTiQqNVJNTSRYIUobKEJZYWhvbyE=?=
=?ISO-2022-JP?B?GyRCJVwlQyUvJTkzK0BfJEskRCQkJEYhSxsoQg==?=
Content-Type:multipart/alternative; boundary="pzu8t3naw7b5therma1t"

--pzu8t3naw7b5therma1t
Content-Type: text/plain; charset=ISO-2022-JP
Content-Transfer-Encoding: 7bit

--pzu8t3naw7b5therma1t
Content-Type: text/html; charset=ISO-2022-JP
Content-Transfer-Encoding: 7bit

--pzu8t3naw7b5therma1t--
```

The code prior to this patch would throw an Error if trailing whitespace was
detected following an ATOM in the context of a LIST or a SECTION. I've
loosened this restriction by removing it entirely for both cases and added test
coverage to match this intentional decision. Note that I don't have any
rationale for letting the much more strict SECTION case have trailing white
space other than it also seems harmless there as well.

There are some similar other restrictions in this place that I have not
loosened.

For future analysis, the server greeting of yahoo.co.jp at this time is:
```
* OK [CAPABILITY IMAP4rev1 ID NAMESPACE UIDPLUS LITERAL+ CHILDREN XAPPLEPUSHSERVICE AUTH=PLAIN AUTH=LOGIN] IMAP4rev1 imapgate-0.7.68_11_1.61475
```