Bug: When moving multiple messages to an IMAP server, mark as deleted each individual successfully copied messages
Status: RESOLVED
Priority: --
Resolution: FIXED
Created: 2023-04-17 05:23:12+00:00
Resolved: 2025-02-07 07:20:02+00:00

+++ This bug was initially created as a clone of Bug #538375: "When attempting to copy or move large numbers of local messages to an IMAP server, it fails without any error or status message. Related to timeouts" +++

Currently when multiple messages are "moved" to an imap folder from a different account (i.e., another imap account, pop3 account or from a Local Folder) each message is sequentially read from the the source folder into a temp file and the file is sent to the destination folder on the server using imap APPEND command triggered by "appendmsgfromfile" imap URL. Once all the message have been copied, they are, only then, marked deleted to complete the "move".

My proposed change is to mark each "moved" message as deleted immediately after the individual message has been successfully copied to the destination folder. This way, after a network or imap server error causes the move to stop, you can tell which messages were copied successfully in the source folder since they are either gone or, if delete method "just mark as deleted" is set, they are crossed out.  Then the move can be restarted by selecting the not yet moved messages and again move the remaining messages to the desired destination.

Of course, doing the multi-message move by copying all the messages and then deleting all the source messages in one command is faster, which is probably why the code currently does that since there will be much fewer round-trips to the server. So it may be possible to preserve this optimization as the default behavior and enable my proposed change via a pref. But I doubt that the speed is that important if a user moves only a few messages, which is typical. Note also that multiple message copies won't be affected by my proposed change since the source messages are not deleted.

Testing with yahoo as the destination imap server, it appears that about 5000 messages can be moved to the server before it complains with a ```* NO [LIMIT] Rate limit hit``` error which terminates the transfer. This is a typical response by yahoo when it thinks TB is abusing it. I've seen this error also sometimes when TB tries to make more than 3 imap connections to yahoo server. It really has nothing to do with commands coming in at a too fast rate.

It's also possible to drag a folder from account A  and drop it to a folder on account B (or to account B itself). Currently when this is done between accounts/servers, the transfer is forced to be just a copy (the source folder(s) and messages remain untouched) so moves are not possible which doesn't help with this bug. Also, drag/drop of folder(s) is the only way to do a recursive folder copy so that a folder's sub-folders and their message are also brought along by the copy.

To fix this so that recursive move of all the messages in the folder and its sub-folders is possible I made a change so that if a move during the drag is detected (e.g., user has not held down ctrl key) still after each message is copied successfully it is marked deleted on the source folder but the source folder and its sub-folders remain present.  So on a successful "move" using drag/drop, the source folders are empty of messages (or contain all marked as deleted / crossed-out messages). Effectively the change copies the folder structure to the destination while moving only the messages.

Note: the reason that drags between accounts only do a copy and never a move has something to do the possible effect on filters, but don't know the details. My proposed change doesn't affect the source folder structure (only deletes the messages) so it shouldn't cause a problem for filters. (But there may be unit test that proves me wrong.)

There is another case where an imap folder is dragged to a Local Folder instead of to another imap account/server. With the current code, again this does a pure copy of folders and messages into the destination Local Folder. So it's not possible to tell which messages may have failed to copy since they are never deleted from the imap source folder.  With my proposed change, each message in the source imap folder is deleted only after a successful file copy to Local Folder occurs. This most typically will result in messages remaining present in the source imap folder when a network error prevents imap fetching of the messages to be written to the destination Local Folder. (Note: this is probably less of an issue when source imap folder uses offline store since, in that case, the transfer is just file to file with no network access required.)

Attached is my preliminary changes required.