Bug: Autoconfig doesn't properly wait for server responses
Status: ASSIGNED
Priority: --
Created: 2020-07-29 23:41:38+00:00
Resolved: 2021-05-27 00:22:22+00:00

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0




Expected results:

Hello Thunderbird team,

While investigating the functionality of the autoconfig engine, we at Linuxmagic noticed another potential issue.

The autoconfig engine, when probing a server, uses a message pump for sending and receiving of protocol commands. Unfortunately, the order of sent and received messages is offset by one due to the logic of the pump handlers.

This off-by-one communication results in what's often known as a "fast talker", where the connected client is sending consecutive commands without waiting for the entire server response first.

A continuous stream of commands is supported by SMTP servers which advertise the PIPELINING keyword... However Tbird sends the EHLO to SMTP immediately before the greeting banner and long before it could ever determine whether PIPELINING may be advertised by the server.

It's not uncommon of for servers to detect and flag "fast talkers" because it often indicates the client was poorly designed or most likely is not a real email client -- often times fast talkers are bots or scripts aiming to spam or bruteforce authentication credentials with no regard for protocol rules. Occasionally fast talkers are devices like printers or IOT devices that lack sophisticated SMTP/email capability, but that is an edge case and obviously those devices are misbehaving.

There is two separate issues in the code which each can trigger fast talker behaviour, only the first problem is actively causing the fast talker, but the second problem *could* become a source of fast talking under certain circumstances (I will explain below).

The two issues which can trigger fast talker behaviour are:

 1. The autoconfig engine does not wait for the server greeting banner

    Immediately at connection Tbird sends the EHLO, without waiting for the greeting banner from the server. This will *always* trigger a fast talker detection and is the main proponent of this issue.

 2. The autoconfig engine does not detect the 'last line of responses'

    This does not immediately cause fast talker issues, however, if for some reason the server greeting banner, or the response to EHLO, were to arrive in two separate packets then Tbird would respond to the first as if it was the entire server message. For this reason protocols like SMTP have a way to identify the last line of a message, we should be identifying this line and only responding to the server once we have encountered the final line.

The current incarnation of the autoconfig engine produces a probe that looks like this (fast talker):

 *connection initialized*

    C: EHLO we-guess.mozilla.org        # fast talker here before greeting banner

    S: 220-localhost ESMTP
       220-MagicMail Daemon with Built-In Spam Protection
       220 See http://www.magicmail.com for more info

    C: QUIT                             # fast talker here before ehlo response

    S: 250-localhost
       250-AUTH LOGIN PLAIN
       250-AUTH=LOGIN PLAIN
       250-STARTTLS
       250-SIZE 10485760
       250-HELP
       250 8BITMIME

    S: 221 localhost

The above transaction is offset by one message, the client has not received the response of EHLO when it sends the QUIT command, and it has not received the greeting banner when it sends the EHLO command.

If Tbird were to wait for the greeting banner first before sending the EHLO, the connection would look something like this:

 *connection initialized*

    S: 220-localhost ESMTP
       220-MagicMail Daemon with Built-In Spam Protection
       220 See http://www.magicmail.com for more info

    C: EHLO we-guess.mozilla.org        # EHLO waited for greeting banner

    S: 250-localhost
       250-AUTH LOGIN PLAIN
       250-AUTH=LOGIN PLAIN
       250-STARTTLS
       250-SIZE 10485760
       250-HELP
       250 8BITMIME

    C: QUIT                             # QUIT waited for EHLO response

    S: 221 localhost

If the remote server sent the greeting banner (or the ehlo response) in two separate packets instead of one then Tbird would respond to the first packet as if it was the entire server message.

That is, Tbird could receive the first *HALF* of the EHLO response, something like:

    S: 250-localhost                                                            
       250-AUTH LOGIN PLAIN                                                     
       250-AUTH=LOGIN PLAIN 


And it would send the QUIT command thinking it received the entire EHLO. 

Of course this is somewhat unlikely... But it's not impossible, especially with the growing number of SMTP services out there.

Both of these issues appear relatively trivial to solve.

-------------------------------------------------------------------------------

The first problem is the onStartRequest() handler of the message pump is sending the EHLO immediately after connecting. If this initial EHLO sending code is simply removed it will fix the first issue entirely by allowing the engine to always behave reactively to server messages instead of pre-emptively sending commands and offsetting the communications.

The second problem is the onData handler is naively responding to every message as if the entire message was received.

In order to alleviate the second issue, at least for SMTP, Tbird could wait and detect the last line of the SMTP responses -- that is a line which starts with three digits and then a space.

All of the lines before the last line will start with three digits and then a hyphen, and the last line will have a space instead of a hyphen, as seen here:

    S: 250-localhost                                                            
       250-AUTH LOGIN PLAIN                                                     
       250-AUTH=LOGIN PLAIN                                                     
       250-STARTTLS                                                             
       250-SIZE 10485760                                                        
       250-HELP  
       250 8BITMIME           # last line has a space after 250

Only after receiving this line should Tbird send the next command in the probe.

To solve this I used a relatively naive approach of adding logic to the function onDataAvailable() which would only respond if the message contains the regex:

  /\d{3} /

Of course this regex is somewhat weak but it's the simplest solution I could come up with. Other suggestions here would probably be good.

--------------------------------------------

Attached is the patch that solves the fast talker issues, the main solution is the removal of the top block of code, and the secondary solution for the secondary issue is the chunk in the onDataHandler which will return if the current protocol being handled is SMTP And the data that is available doesn't contain the regex /\d{3} /