Bug: Flaky TestPaxos
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/May/21 08:21
Resolved: 19/May/21 13:17

During testing for some other ticket I found in a test run [this|https://ci-cassandra.apache.org/job/Cassandra-devbranch/736/testReport/junit/dtest-novnode.paxos_test/TestPaxos/test_cluster_availability/] paxos failure

{noformat}
Error Message

cassandra.Unavailable: Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level SERIAL" info={'consistency': 'SERIAL', 'required_replicas': 1, 'alive_replicas': 0}

Stacktrace

self = <paxos_test.TestPaxos object at 0x7fafbd3f4850>

    @pytest.mark.no_vnodes
    def test_cluster_availability(self):
        # Warning, a change in partitioner or a change in CCM token allocation
        # may require the partition keys of these inserts to be changed.
        # This must not use vnodes as it relies on assumed token values.
    
        session = self.prepare(nodes=3)
        session.execute("CREATE TABLE test (k int PRIMARY KEY, v int)")
        session.execute("INSERT INTO test (k, v) VALUES (0, 0) IF NOT EXISTS")
    
        self.cluster.nodelist()[2].stop()
        session.execute("INSERT INTO test (k, v) VALUES (1, 1) IF NOT EXISTS")
    
        self.cluster.nodelist()[1].stop()
        session.execute("INSERT INTO test (k, v) VALUES (3, 2) IF NOT EXISTS")
    
        self.cluster.nodelist()[1].start()
        session.execute("INSERT INTO test (k, v) VALUES (5, 5) IF NOT EXISTS")
    
        self.cluster.nodelist()[2].start()
>       session.execute("INSERT INTO test (k, v) VALUES (6, 6) IF NOT EXISTS")

paxos_test.py:83: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../venv/src/cassandra-driver/cassandra/cluster.py:2618: in execute
    return self.execute_async(query, parameters, trace, custom_payload, timeout, execution_profile, paging_state, host, execute_as).result()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ResponseFuture: query='<SimpleStatement query="INSERT INTO test (k, v) VALUES (6, 6) IF NOT EXISTS", consistency=Not ...el SERIAL" info={'consistency': 'SERIAL', 'required_replicas': 1, 'alive_replicas': 0} coordinator_host=127.0.0.2:9042>

    def result(self):
        """
            Return the final result or raise an Exception if errors were
            encountered.  If the final result or error has not been set
            yet, this method will block until it is set, or the timeout
            set for the request expires.
    
            Timeout is specified in the Session request execution functions.
            If the timeout is exceeded, an :exc:`cassandra.OperationTimedOut` will be raised.
            This is a client-side timeout. For more information
            about server-side coordinator timeouts, see :class:`.policies.RetryPolicy`.
    
            Example usage::
    
                >>> future = session.execute_async("SELECT * FROM mycf")
                >>> # do other stuff...
    
                >>> try:
                ...     rows = future.result()
                ...     for row in rows:
                ...         ... # process results
                ... except Exception:
                ...     log.exception("Operation failed:")
    
            """
        self._event.wait()
        if self._final_result is not _NOT_SET:
            return ResultSet(self, self._final_result)
        else:
>           raise self._final_exception
E           cassandra.Unavailable: Error from server: code=1000 [Unavailable exception] message="Cannot achieve consistency level SERIAL" info={'consistency': 'SERIAL', 'required_replicas': 1, 'alive_replicas': 0}

../venv/src/cassandra-driver/cassandra/cluster.py:4894: Unavailable
{noformat}