email: fixed missing port argument for IMAPAuthenticationError

Signed-off-by: Arun Persaud <apersaud@lbl.gov>
CHANGELOG: Document missing port argument for IMAPAuthenticationError fix

Documenting:
9124d62 email: fixed missing port argument for IMAPAuthenticationError

Signed-off-by: W. Trevor King <wking@tremily.us>
Bump to version 3.6

Signed-off-by: W. Trevor King <wking@tremily.us>
feed: Disable feedparser's PREFERRED_XML_PARSERS

Feedparser's default parser (drv_libxml2) has trouble parsing byte
streams in Python 3:

  $ python -c 'import rss2email.feed; import doctest; doctest.testmod(rss2email.feed)'
  ...
  File "rss2email/feed.py", line 319, in rss2email.feed.Feed._fetch
Failed example:
    parsed = feed._fetch()
Exception raised:
    Traceback (most recent call last):
      File "rss2email/util.py", line 61, in run
        self.result = self._target(*self._args, **self._kwargs)
      File "/.../feedparser/feedparser.py", line 3745, in parse
        saxparser.parse(source)
      File "/usr/lib64/python3.2/site-packages/drv_libxml2.py", line 270, in parse
        _d(reader.Name()),_d(reader.Value()))
      File "/usr/lib64/python3.2/site-packages/drv_libxml2.py", line 70, in _d
        return _decoder(s)[0]
      File "/usr/lib64/python3.2/encodings/utf_8.py", line 16, in decode
        return codecs.utf_8_decode(input, errors, True)
    TypeError: 'str' does not support the buffer interface

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "/usr/lib64/python3.2/doctest.py", line 1288, in __run
        compileflags, 1), test.globs)
      File "<doctest rss2email.feed.Feed._fetch[1]>", line 1, in <module>
        parsed = feed._fetch()
      File "rss2email/feed.py", line 336, in _fetch
        return f(self.url, self.etag, modified=self.modified, **kwargs)
      File "rss2email/util.py", line 76, in __call__
        time_limited_function=self) from self.error[1]
    rss2email.error.TimeoutError: error while running time limited function: 'str' does not support the buffer interface
  ...

You can reproduce the underlying exception with this minimal script:

  import io
  import xml.sax
  import xml.sax.handler

  data = b'<feed xmlns="http://www.w3.org/2005/Atom"><entry><author><name>Example author</name><email>me@example.com</email><url>http://example.com/</url></author></entry></feed>'

  source = xml.sax.xmlreader.InputSource()
  source.setByteStream(io.BytesIO(data))
  saxparser = xml.sax.make_parser(["drv_libxml2"])
  saxparser.setContentHandler(xml.sax.handler.ContentHandler())
  saxparser.parse(source)

which raises:

  Traceback (most recent call last):
    File "<stdin>", line 13, in <module>
      saxparser.parse(source)
    File "/usr/lib64/python3.2/site-packages/drv_libxml2.py", line 222, in parse
      eltName = _d(reader.Name())
    File "/usr/lib64/python3.2/site-packages/drv_libxml2.py", line 70, in _d
      return _decoder(s)[0]
    File "/usr/lib64/python3.2/encodings/utf_8.py", line 16, in decode
      return codecs.utf_8_decode(input, errors, True)
  TypeError: 'str' does not support the buffer interface

at least for libxml2-2.9.1.

By using the stdlib's default parser (instead of drv_libxml2), we can
avoid the error and get successful parsing.  If you don't have
drv_libxml2 installed, sax was already falling back on the stdlib's
default parser, so this commit will be a no-op.

Signed-off-by: W. Trevor King <wking@tremily.us>
feed: Convert 'friendly-name' setting to 'name-format'

In Debian bug 722009, Joey Hess wrote about the 2.x series [1]:
> The current From line generated by r2e is From: Blog: Author
>   where
>     Blog is the name of the blog, or some page like a wiki's RecentChanges
>     Author is the author of a post
>       iff the blog sets that info in the feed
>
> In mutt the Blog part often occupies the whole displayed Subject field,
> which is fixed width. So the Author cannot be seen. This is particularly
> a problem with planets, where the author of a post matters a lot.
> But also with some blogs that have multiple authors.
>
> For these sorts of blogs, I would generally prefer to use a From line
> like From: Author (Blog)
> This does mean that when sorting by author, all posts of a blog or
> planet feed may not appear together, but that would be an acceptable
> tradeoff to me.
>
> One way to implement this (other than just changing the format string)
> would be to make OVERRIDE_FROM able to contain a format string,
> so it could be configured on a per-feed basis.

The new setup makes the name-formatting configurable on a per feed
basis (what Joey wanted), but it's not without side effects.  For
feeds where some information is missing (feed-title, author, or
publisher), we used to adjust the formatting on the fly.  For example,
you'd get output like '{author}' if the feed-title was missing,
instead of getting ': {author}'.  Now users bothered by this will have
to manually override the format template for feeds missing crucial
data.

[1]: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=722009

Signed-off-by: W. Trevor King <wking@tremily.us>
feed: don't emit error if parser able to auto-determine encoding

Some feeds have incorrectly declared encodings (e.g. the encoding
specified by the HTTP header does not match the encoding specified in
the XML declaration).  For such a feed, "r2e run" would emit an error
message similar to the following:

  processing error: document declared as us-ascii, but parsed as
  iso-8859-1: undeadly (http://undeadly.org/cgi?action=rss ->
  jlmuir@imca-cat.org)

In this particular case, the HTTP header indicated a content type of
"text/xml" with no "charset" parameter.  According to the feedparser
5.1.3 documentation (section "Introduction to Character Encoding" [1]),
this results in an encoding of US-ASCII.  But the served XML document
contains an encoding declaration of ISO-8859-1.

For this case and some others, feedparser is able to automatically
determine an encoding.  When it does, we emit a warning rather than an
error, and accept the automatically determined encoding.

We check for a successfully overridden encoding by looking at the bozo
bit and the bozo_exception.  If the bozo bit is set and the
bozo_exception is feedparser.CharacterEncodingOverride, the parser has
successfully overridden an incorrectly declared encoding.  Quoting from
the feedparser 5.1.3 documentation, section "Handling
Incorrectly-Declared Encodings" [2]:

  Universal Feed Parser initially uses the rules specified in RFC 3023
  to determine the character encoding of the feed. If parsing succeeds,
  then that's that. If parsing fails, Universal Feed Parser sets the
  bozo bit to 1 and sets bozo_exception to
  feedparser.CharacterEncodingOverride. Then it tries to reparse the
  feed with the following character encodings:

  1. the encoding specified in the XML declaration
  2. the encoding sniffed from the first four bytes of the document (as
     per Section F)
  3. the encoding auto-detected by the Universal Encoding Detector, if
     installed
  4. utf-8
  5. windows-1252

  If the character encoding can not be determined, Universal Feed Parser
  sets the bozo bit to 1 and sets bozo_exception to
  feedparser.CharacterEncodingUnknown. In this case, parsed values will
  be strings, not Unicode strings.

References:

1. http://pythonhosted.org/feedparser/character-encoding.html#introduction-to-character-encoding
2. http://pythonhosted.org/feedparser/character-encoding.html#handling-incorrectly-declared-encodings

Signed-off-by: J. Lewis Muir <jlmuir@imca-cat.org>
CHANGELOG: Demote guessed encodings logs from 'error' to 'warning'

Documenting:
05f2628 feed: don't emit error if parser able to auto-determine encoding

Signed-off-by: W. Trevor King <wking@tremily.us>
Merge branch 'accept-feedparser-encoding-override'

* accept-feedparser-encoding-override:
  CHANGELOG: Demote guessed encodings logs from 'error' to 'warning'
  feed: don't emit error if parser able to auto-determine encoding

Signed-off-by: W. Trevor King <wking@tremily.us>
Merge branch 'no-prefered-xml-parser'

* no-prefered-xml-parser:
  feed: Disable feedparser's PREFERRED_XML_PARSERS

Signed-off-by: W. Trevor King <wking@tremily.us>
feed: Give defaults for _get_entry_name formatting data

We don't want to crash if the source feed is missing some data that
the user expects, or if the user just hasn't had the time to adjust
the name-format config.

Signed-off-by: W. Trevor King <wking@tremily.us>
Merge branch 'format-entry-name'

* format-entry-name:
  feed: Give defaults for _get_entry_name formatting data
  feed: Convert 'friendly-name' setting to 'name-format'

Signed-off-by: W. Trevor King <wking@tremily.us>
Run update-copyright.py

Signed-off-by: W. Trevor King <wking@tremily.us>
r2e.1: Remove quotes around 'name-format' value

ConfigParser doesn't need quoting around string values, so if you use
quotes they will show up explicitly:

  From: "'indexed: Jessica Hagy'" <user@rss2email.invalid>

That's probably not what you want ;).

Signed-off-by: W. Trevor King <wking@tremily.us>
feeds: Make Feeds.save fully atomic, assuming a working fsync

If the disk is full (or there are other OS-level issues), a file may
not be completely written to the disk.

The write-flush-fsync-rename sequence is much safer.  The fsync
invocation matches the recommendation in the docs [1]:

  If you’re starting with a buffered Python file object f, first do
  f.flush(), and then do os.fsync(f.fileno()), to ensure that all
  internal buffers associated with f are written to disk.

The purpose of each step is:

* write: move the data into a library buffer
* flush: flush the library buffer into a kernel buffer
* fsync: flush the kernel buffer onto the disk at $tempfile
* rename: adjust the metadata so that the $filename points to the
    $tempfile data, release the old data

This means that if the rename works we get the new data, and if the
rename fails we still have the old data.

However, POSIX's fsync is implementation defined unless
_POSIX_SYNCHRONIZED_IO is defined [3,4], and some OS X implementations
go the no-op route, as Stewart Smith points out in his excellent "Eat
My Data: How everybody gets file I/O wrong" [4].  If you want to run
rss2email on such a system, verifying your data integrity is up to you
;).

We used to write-rename the data file (but not the config) on *nix
[2].  Now we do the full write-flush-fsync-rename for both the config
and data files on both *nix and other systems.

[1]: http://docs.python.org/3/library/os.html#os.fsync
[2]: For rss2email, *nix is "has fcntl, but isn't SunOS
[3]: http://pubs.opengroup.org/onlinepubs/009695399/functions/fsync.html
[4]: https://www.flamingspork.com/talks/2007/06/eat_my_data.odp

Reported-by: Etienne Millon <me@emillon.org>
Signed-off-by: W. Trevor King <wking@tremily.us>
feed: Add 'default' argument to Feed._html2text for HTMLParseError

This allows us to easily fall back on an unconverted string in the
event that the input HTML is malformed.  We already caught
HTMLParseError when converting HTML to plain test for non-html mail,
but we didn't catch it in Feed._get_entry_title.  Now we gracefully
handle the situation by treating the malformed HTML as plain text.
CHANGELOG: Document this branch's atomic saves

Signed-off-by: W. Trevor King <wking@tremily.us>
Merge branch 'robust-file-saving'

* robust-file-saving:
  CHANGELOG: Document this branch's atomic saves
  feeds: Make Feeds.save fully atomic, assuming a working fsync

Signed-off-by: W. Trevor King <wking@tremily.us>
CHANGELOG: Document this branch's HTML-title conversion fallback

Signed-off-by: W. Trevor King <wking@tremily.us>
Merge branch 'catch-html-parse-error'

* catch-html-parse-error:
  CHANGELOG: Document this branch's HTML-title conversion fallback
  feed: Add 'default' argument to Feed._html2text for HTMLParseError

Signed-off-by: W. Trevor King <wking@tremily.us>
README: Three-space indents for nested enumerated lists

Apparently two spaces doesn't cut it.  This change fixes:

  $ rst2html.py --strict README
  README:5: (INFO/1) Enumerated list start value not ordinal-1: "3" (ordinal 3)
  Exiting due to level-1 (INFO) system message.

I also use a bullet list for dependencies, because the order in which
you install them doesn't matter to me.

Signed-off-by: W. Trevor King <wking@tremily.us>