Repository: getmoto/moto

Issue:
botocore.parsers.ResponseParserError changing visibility on message not in flight
How to reproduce the bug:

Run moto docker in server mode (I used the latest image at https://hub.docker.com/r/motoserver/moto)

Create a queue, send a message, receive the message, and try to set the visibility on the (now no longer valid) message:

`queue = boto3.resource('sqs', endpoint_url='http://moto:5000').create_queue(QueueName='test')`
`msg = base64.b64encode(json.dumps({'test': 'test'}).encode('utf-8')).decode('utf-8')`
`queue.send_message(MessageBody=msg)`
`sqs_msg = queue.receive_messages()[0]`
`sqs_msg.change_visibility(VisibilityTimeout=1)`

Expected:
`botocore.exceptions.ClientError`

What happens:
`*** botocore.parsers.ResponseParserError: Unable to parse response (syntax error: line 1, column 0), invalid XML received. Further retries may succeed:
b'Invalid request: The message referred to is not in flight.'`

Possible cause:
MessageNotInflight is the only exception in sqs/exceptions.py that does not inherit from RESTError. If I do make it inherit from RESTError, the issue goes away.

Version: Server mode using https://hub.docker.com/r/motoserver/moto latest (digest ef4e4172daf2) using docker-compose on the default port (5000).


Discussion:
Thanks for raising this, @wallowa!
Hi @wallowa, do you have a script to reproduce this behaviour against AWS itself? 
Executing the code in the OP gives me a standard 200 response, i.e. AWS allows changing the visibility of a message in flight.
Doing further investigation
Comparing AWS behavior to moto.

**Change_visibility after a message has become visible again**

AWS
```
In [47]: x.change_visibility(VisibilityTimeout=0)
Out[47]: 
{'ResponseMetadata': {...
  'HTTPStatusCode': 200,
  ...}}

In [48]: x.change_visibility(VisibilityTimeout=0)
Out[48]: 
{'ResponseMetadata': {...
  'HTTPStatusCode': 200,
  ...}}
```
moto:
```
(Pdb) x = queue.receive_messages()[0]
(Pdb) x.change_visibility(VisibilityTimeout=0)
{'ResponseMetadata': {'RequestId': '1U0WFGZUMKRHBNW3SHCMD6H8IAVTEXI5Q0K0F2L7AR82NKH2QWIO', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com', 'x-amzn-requestid': '1U0WFGZUMKRHBNW3SHCMD6H8IAVTEXI5Q0K0F2L7AR82NKH2QWIO', 'x-amz-crc32': '3375092054', 'content-type': 'text/html; charset=utf-8', 'content-length': '179', 'date': 'Wed, 10 Mar 2021 19:28:11 GMT'}, 'RetryAttempts': 0}}
(Pdb) x.change_visibility(VisibilityTimeout=0)
*** botocore.parsers.ResponseParserError: Unable to parse response (syntax error: line 1, column 0), invalid XML received. Further retries may succeed:
b'Invalid request: The message referred to is not in flight.'
```
This shows two behaviors that are different:
1. There appears to be a bug in moto's MessageNotInflight class.
2. Moto does not appear to allow calling change_visibility on a message that's become visible again, though AWS does.

The class is currently
```
class MessageNotInflight(Exception):
    description = "The message referred to is not in flight."
    status_code = 400
```
But it should be:
```
    class MessageNotInflight(RESTError):
        code = 400

        def __init__(self):
            super().__init__(
                "MessageNotInflight", "Invalid request: The message referred to is not in flight."
            )
```

**Multiple deletes on a message**

AWS
```
In [49]: x.delete()
Out[49]: 
{'ResponseMetadata': {...
  'HTTPStatusCode': 200,
  ...}}

In [50]: x.delete()
Out[50]: 
{'ResponseMetadata': {...
  'HTTPStatusCode': 200,
  ...}}
```
Moto:
```
(Pdb) x = queue.receive_messages()[0]
(Pdb) x.delete()
{'ResponseMetadata': {'RequestId': 'Z1G1EYA4BFKB24XBTNN8Q3W27DPW9I7DE2QXESQ6V2I30RET72MB', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com', 'x-amzn-requestid': 'Z1G1EYA4BFKB24XBTNN8Q3W27DPW9I7DE2QXESQ6V2I30RET72MB', 'x-amz-crc32': '1556951917', 'content-type': 'text/html; charset=utf-8', 'content-length': '159', 'date': 'Wed, 10 Mar 2021 19:28:57 GMT'}, 'RetryAttempts': 0}}
(Pdb) x.delete()
*** botocore.errorfactory.ReceiptHandleIsInvalid: An error occurred (ReceiptHandleIsInvalid) when calling the DeleteMessage operation: The input receipt handle is invalid.
```



Resolving patch:
diff --git a/moto/sqs/exceptions.py b/moto/sqs/exceptions.py
--- a/moto/sqs/exceptions.py
+++ b/moto/sqs/exceptions.py
@@ -1,11 +1,6 @@
 from moto.core.exceptions import RESTError
 
 
-class MessageNotInflight(Exception):
-    description = "The message referred to is not in flight."
-    status_code = 400
-
-
 class ReceiptHandleIsInvalid(RESTError):
     code = 400
 
diff --git a/moto/sqs/models.py b/moto/sqs/models.py
--- a/moto/sqs/models.py
+++ b/moto/sqs/models.py
@@ -24,7 +24,6 @@
 from .utils import generate_receipt_handle
 from .exceptions import (
     MessageAttributesInvalid,
-    MessageNotInflight,
     QueueDoesNotExist,
     QueueAlreadyExists,
     ReceiptHandleIsInvalid,
@@ -73,6 +72,7 @@ def __init__(self, message_id, body, system_attributes={}):
         self._body = body
         self.message_attributes = {}
         self.receipt_handle = None
+        self._old_receipt_handles = []
         self.sender_id = DEFAULT_SENDER_ID
         self.sent_timestamp = None
         self.approximate_first_receive_timestamp = None
@@ -178,6 +178,7 @@ def mark_received(self, visibility_timeout=None):
         if visibility_timeout:
             self.change_visibility(visibility_timeout)
 
+        self._old_receipt_handles.append(self.receipt_handle)
         self.receipt_handle = generate_receipt_handle()
 
     def change_visibility(self, visibility_timeout):
@@ -203,6 +204,16 @@ def delayed(self):
             return True
         return False
 
+    @property
+    def all_receipt_handles(self):
+        return [self.receipt_handle] + self._old_receipt_handles
+
+    def had_receipt_handle(self, receipt_handle):
+        """
+        Check if this message ever had this receipt_handle in the past
+        """
+        return receipt_handle in self.all_receipt_handles
+
 
 class Queue(CloudFormationModel):
     BASE_ATTRIBUTES = [
@@ -247,6 +258,7 @@ def __init__(self, name, region, **kwargs):
 
         self._messages = []
         self._pending_messages = set()
+        self.deleted_messages = set()
 
         now = unix_time()
         self.created_timestamp = now
@@ -541,6 +553,26 @@ def add_message(self, message):
                     for m in messages
                 ]
 
+    def delete_message(self, receipt_handle):
+        if receipt_handle in self.deleted_messages:
+            # Already deleted - gracefully handle deleting it again
+            return
+
+        if not any(
+            message.had_receipt_handle(receipt_handle) for message in self._messages
+        ):
+            raise ReceiptHandleIsInvalid()
+
+        # Delete message from queue regardless of pending state
+        new_messages = []
+        for message in self._messages:
+            if message.had_receipt_handle(receipt_handle):
+                self.pending_messages.discard(message)
+                self.deleted_messages.update(message.all_receipt_handles)
+                continue
+            new_messages.append(message)
+        self._messages = new_messages
+
     @classmethod
     def has_cfn_attr(cls, attribute_name):
         return attribute_name in ["Arn", "QueueName"]
@@ -906,26 +938,12 @@ def receive_messages(
     def delete_message(self, queue_name, receipt_handle):
         queue = self.get_queue(queue_name)
 
-        if not any(
-            message.receipt_handle == receipt_handle for message in queue._messages
-        ):
-            raise ReceiptHandleIsInvalid()
-
-        # Delete message from queue regardless of pending state
-        new_messages = []
-        for message in queue._messages:
-            if message.receipt_handle == receipt_handle:
-                queue.pending_messages.discard(message)
-                continue
-            new_messages.append(message)
-        queue._messages = new_messages
+        queue.delete_message(receipt_handle)
 
     def change_message_visibility(self, queue_name, receipt_handle, visibility_timeout):
         queue = self.get_queue(queue_name)
         for message in queue._messages:
-            if message.receipt_handle == receipt_handle:
-                if message.visible:
-                    raise MessageNotInflight
+            if message.had_receipt_handle(receipt_handle):
 
                 visibility_timeout_msec = int(visibility_timeout) * 1000
                 given_visibility_timeout = unix_time_millis() + visibility_timeout_msec
@@ -938,7 +956,7 @@ def change_message_visibility(self, queue_name, receipt_handle, visibility_timeo
                     )
 
                 message.change_visibility(visibility_timeout)
-                if message.visible:
+                if message.visible and message in queue.pending_messages:
                     # If the message is visible again, remove it from pending
                     # messages.
                     queue.pending_messages.remove(message)
diff --git a/moto/sqs/responses.py b/moto/sqs/responses.py
--- a/moto/sqs/responses.py
+++ b/moto/sqs/responses.py
@@ -14,7 +14,6 @@
     EmptyBatchRequest,
     InvalidAddress,
     InvalidAttributeName,
-    MessageNotInflight,
     ReceiptHandleIsInvalid,
     BatchEntryIdsNotDistinct,
 )
@@ -123,17 +122,11 @@ def change_message_visibility(self):
         except ValueError:
             return ERROR_MAX_VISIBILITY_TIMEOUT_RESPONSE, dict(status=400)
 
-        try:
-            self.sqs_backend.change_message_visibility(
-                queue_name=queue_name,
-                receipt_handle=receipt_handle,
-                visibility_timeout=visibility_timeout,
-            )
-        except MessageNotInflight as e:
-            return (
-                "Invalid request: {0}".format(e.description),
-                dict(status=e.status_code),
-            )
+        self.sqs_backend.change_message_visibility(
+            queue_name=queue_name,
+            receipt_handle=receipt_handle,
+            visibility_timeout=visibility_timeout,
+        )
 
         template = self.response_template(CHANGE_MESSAGE_VISIBILITY_RESPONSE)
         return template.render()
@@ -176,15 +169,6 @@ def change_message_visibility_batch(self):
                         "Message": e.description,
                     }
                 )
-            except MessageNotInflight as e:
-                error.append(
-                    {
-                        "Id": entry["id"],
-                        "SenderFault": "false",
-                        "Code": "AWS.SimpleQueueService.MessageNotInflight",
-                        "Message": e.description,
-                    }
-                )
 
         template = self.response_template(CHANGE_MESSAGE_VISIBILITY_BATCH_RESPONSE)
         return template.render(success=success, errors=error)


Test patch:
diff --git a/tests/test_sqs/test_sqs.py b/tests/test_sqs/test_sqs.py
--- a/tests/test_sqs/test_sqs.py
+++ b/tests/test_sqs/test_sqs.py
@@ -1677,6 +1677,23 @@ def test_change_message_visibility_boto3():
     queue.attributes["ApproximateNumberOfMessages"].should.equal("0")
 
 
+@mock_sqs
+def test_change_message_visibility_on_unknown_receipt_handle():
+    sqs = boto3.resource("sqs", region_name="us-east-1")
+    conn = boto3.client("sqs", region_name="us-east-1")
+    queue = sqs.create_queue(
+        QueueName=str(uuid4())[0:6], Attributes={"VisibilityTimeout": "2"}
+    )
+
+    with pytest.raises(ClientError) as exc:
+        conn.change_message_visibility(
+            QueueUrl=queue.url, ReceiptHandle="unknown-stuff", VisibilityTimeout=432,
+        )
+    err = exc.value.response["Error"]
+    err["Code"].should.equal("ReceiptHandleIsInvalid")
+    err["Message"].should.equal("The input receipt handle is invalid.")
+
+
 # Has boto3 equivalent
 @mock_sqs_deprecated
 def test_message_attributes():
@@ -1887,39 +1904,11 @@ def test_queue_attributes():
     attribute_names.should.contain("QueueArn")
 
 
-# Has boto3 equivalent
-@mock_sqs_deprecated
-def test_change_message_visibility_on_invalid_receipt():
-    conn = boto.connect_sqs("the_key", "the_secret")
-    queue = conn.create_queue("test-queue", visibility_timeout=1)
-    queue.set_message_class(RawMessage)
-
-    queue.write(queue.new_message("this is another test message"))
-    queue.count().should.equal(1)
-    messages = conn.receive_message(queue, number_messages=1)
-
-    assert len(messages) == 1
-
-    original_message = messages[0]
-
-    queue.count().should.equal(0)
-
-    time.sleep(2)
-
-    queue.count().should.equal(1)
-
-    messages = conn.receive_message(queue, number_messages=1)
-
-    assert len(messages) == 1
-
-    original_message.change_visibility.when.called_with(100).should.throw(SQSError)
-
-
 @mock_sqs
-def test_change_message_visibility_on_invalid_receipt_boto3():
+def test_change_message_visibility_on_old_message_boto3():
     sqs = boto3.resource("sqs", region_name="us-east-1")
     queue = sqs.create_queue(
-        QueueName=str(uuid4())[0:6], Attributes={"VisibilityTimeout ": "1"}
+        QueueName=str(uuid4())[0:6], Attributes={"VisibilityTimeout": "1"}
     )
 
     queue.send_message(MessageBody="test message 1")
@@ -1942,62 +1931,43 @@ def test_change_message_visibility_on_invalid_receipt_boto3():
 
     messages.should.have.length_of(1)
 
-    with pytest.raises(ClientError) as ex:
-        original_message.change_visibility(VisibilityTimeout=100)
-    err = ex.value.response["Error"]
-    err["Code"].should.equal("ReceiptHandleIsInvalid")
-    err["Message"].should.equal("The input receipt handle is invalid.")
-
-
-# Has boto3 equivalent
-@mock_sqs_deprecated
-def test_change_message_visibility_on_visible_message():
-    conn = boto.connect_sqs("the_key", "the_secret")
-    queue = conn.create_queue("test-queue", visibility_timeout=1)
-    queue.set_message_class(RawMessage)
-
-    queue.write(queue.new_message("this is another test message"))
-    queue.count().should.equal(1)
-    messages = conn.receive_message(queue, number_messages=1)
-
-    assert len(messages) == 1
-
-    original_message = messages[0]
-
-    queue.count().should.equal(0)
+    # Docs indicate this should throw an ReceiptHandleIsInvalid, but this is allowed in AWS
+    original_message.change_visibility(VisibilityTimeout=100)
+    # Docs indicate this should throw a MessageNotInflight, but this is allowed in AWS
+    original_message.change_visibility(VisibilityTimeout=100)
 
     time.sleep(2)
 
-    queue.count().should.equal(1)
-
-    original_message.change_visibility.when.called_with(100).should.throw(SQSError)
+    # Message is not yet available, because of the visibility-timeout
+    messages = queue.receive_messages(MaxNumberOfMessages=1)
+    messages.should.have.length_of(0)
 
 
 @mock_sqs
 def test_change_message_visibility_on_visible_message_boto3():
     sqs = boto3.resource("sqs", region_name="us-east-1")
     queue = sqs.create_queue(
-        QueueName=str(uuid4())[0:6], Attributes={"VisibilityTimeout ": "1"}
+        QueueName=str(uuid4())[0:6], Attributes={"VisibilityTimeout": "1"}
     )
 
     queue.send_message(MessageBody="test message")
     messages = queue.receive_messages(MaxNumberOfMessages=1)
     messages.should.have.length_of(1)
 
-    original_message = messages[0]
-
     queue.reload()
     queue.attributes["ApproximateNumberOfMessages"].should.equal("0")
 
     time.sleep(2)
 
-    queue.reload()
-    queue.attributes["ApproximateNumberOfMessages"].should.equal("1")
+    messages = queue.receive_messages(MaxNumberOfMessages=1)
+    messages.should.have.length_of(1)
+
+    messages[0].change_visibility(VisibilityTimeout=100)
 
-    # TODO: We should catch a ClientError here, but Moto throws an error in the wrong format
-    with pytest.raises(Exception) as ex:
-        original_message.change_visibility(VisibilityTimeout=100)
-    str(ex).should.match("Invalid request: The message referred to is not in flight.")
+    time.sleep(2)
+
+    queue.reload()
+    queue.attributes["ApproximateNumberOfMessages"].should.equal("0")
 
 
 # Has boto3 equivalent
@@ -2112,6 +2082,48 @@ def test_delete_message_errors():
     ).should.throw(ClientError, "The input receipt handle is invalid.")
 
 
+@mock_sqs
+def test_delete_message_twice_using_same_receipt_handle():
+    client = boto3.client("sqs", region_name="us-east-1")
+    response = client.create_queue(QueueName=str(uuid4())[0:6])
+    queue_url = response["QueueUrl"]
+
+    client.send_message(QueueUrl=queue_url, MessageBody="body")
+    response = client.receive_message(QueueUrl=queue_url)
+    receipt_handle = response["Messages"][0]["ReceiptHandle"]
+
+    client.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle)
+    client.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle)
+
+
+@mock_sqs
+def test_delete_message_using_old_receipt_handle():
+    client = boto3.client("sqs", region_name="us-east-1")
+    response = client.create_queue(
+        QueueName=str(uuid4())[0:6], Attributes={"VisibilityTimeout": "0"}
+    )
+    queue_url = response["QueueUrl"]
+
+    client.send_message(QueueUrl=queue_url, MessageBody="body")
+    response = client.receive_message(QueueUrl=queue_url)
+    receipt_1 = response["Messages"][0]["ReceiptHandle"]
+
+    response = client.receive_message(QueueUrl=queue_url)
+    receipt_2 = response["Messages"][0]["ReceiptHandle"]
+
+    receipt_1.shouldnt.equal(receipt_2)
+
+    # Can use an old receipt_handle to delete a message
+    client.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_1)
+    # Sanity check the message really is gone
+    client.receive_message(QueueUrl=queue_url).shouldnt.have.key("Messages")
+    # We can delete it again
+    client.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_1)
+
+    # Can use the second receipt handle to delete it 'again' - succeeds, as it is idempotent against the message
+    client.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_2)
+
+
 @mock_sqs
 def test_send_message_batch():
     client = boto3.client("sqs", region_name="us-east-1")
@@ -2421,6 +2433,38 @@ def test_batch_change_message_visibility():
         len(resp["Messages"]).should.equal(3)
 
 
+@mock_sqs
+def test_batch_change_message_visibility_on_old_message():
+    sqs = boto3.resource("sqs", region_name="us-east-1")
+    queue = sqs.create_queue(
+        QueueName=str(uuid4())[0:6], Attributes={"VisibilityTimeout": "1"}
+    )
+
+    queue.send_message(MessageBody="test message 1")
+
+    messages = queue.receive_messages(MaxNumberOfMessages=1)
+
+    messages.should.have.length_of(1)
+
+    original_message = messages[0]
+
+    time.sleep(2)
+
+    messages = queue.receive_messages(MaxNumberOfMessages=1)
+    messages[0].receipt_handle.shouldnt.equal(original_message.receipt_handle)
+
+    entries = [
+        {
+            "Id": str(uuid.uuid4()),
+            "ReceiptHandle": original_message.receipt_handle,
+            "VisibilityTimeout": 4,
+        }
+    ]
+
+    resp = queue.change_message_visibility_batch(Entries=entries)
+    resp["Successful"].should.have.length_of(1)
+
+
 @mock_sqs
 def test_permissions():
     client = boto3.client("sqs", region_name="us-east-1")