Bug: Fix python dtest upgrade tests
Status: Open
Priority: Normal
Created: 29/Oct/21 19:09

Several tests are failing in python dtest upgrade

https://app.circleci.com/pipelines/github/dcapwell/cassandra/1062/workflows/ba3e6395-ef22-4724-8424-0549e65d8cff/jobs/7095

upgrade_tests/drop_compact_storage_upgrade_test.py::TestDropCompactStorage:: drop_compact_storage

{code}
assert 'Cannot DROP COMPACT STORAGE as some nodes in the cluster ([/127.0.0.2:7000, /127.0.0.1:7000]) are not on 4.0+ yet. Please upgrade those nodes and run `upgradesstables` before retrying.' in 'Error from server: code=2200 [Invalid query] message="Cannot DROP COMPACT STORAGE as some nodes in the cluster ([/127....1:7000, /127.0.0.2:7000]) are not on 4.0+ yet. Please upgrade those nodes and run `upgradesstables` before retrying."'  +  where 'Error from server: code=2200 [Invalid query] message="Cannot DROP COMPACT STORAGE as some nodes in the cluster ([/127....1:7000, /127.0.0.2:7000]) are not on 4.0+ yet. Please upgrade those nodes and run `upgradesstables` before retrying."' = str(InvalidRequest('Error from server: code=2200 [Invalid query] message="Cannot DROP COMPACT STORAGE as some nodes in the...:7000, /127.0.0.2:7000]) are not on 4.0+ yet. Please upgrade those nodes and run `upgradesstables` before retrying."',))
{code}


bootstrap_test.py::TestBootstrapUpgrade:: test_bootstrap_with_reset_bootstrap_state

This is likely related to CASSANDRA-17055

{code}
>       node1.start()

bootstrap_test.py:483: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../env3.6/lib/python3.6/site-packages/ccmlib/node.py:895: in start
    node.watch_log_for_alive(self, from_mark=mark)
../env3.6/lib/python3.6/site-packages/ccmlib/node.py:664: in watch_log_for_alive
    self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout, filename=filename)
../env3.6/lib/python3.6/site-packages/ccmlib/node.py:592: in watch_log_for
    head=reads[:50], tail="..."+reads[len(reads)-150:]))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

start = 1635454559.2164426, timeout = 120
msg = "Missing: ['127.0.0.1:7000.* is now UP'] not found in system.log:\n Head: \n Tail: ..."
node = 'node3'

    @staticmethod
    def raise_if_passed(start, timeout, msg, node=None):
        if start + timeout < time.time():
>           raise TimeoutError.create(start, timeout, msg, node)
E           ccmlib.node.TimeoutError: 28 Oct 2021 20:57:59 [node3] after 120.13/120 seconds Missing: ['127.0.0.1:7000.* is now UP'] not found in system.log:
E            Head: 
E            Tail: ...
{code}


upgrade_tests/upgrade_through_versions_test.py::TestProtoV3Upgrade_AllVersions_EndsAt_3_11_X:: test_rolling_upgrade

{code}
>           raise RuntimeError(message)
E           RuntimeError: A subprocess has terminated early. Subprocess statuses: Process-1 (is_alive: True), Process-2 (is_alive: False), attempting to terminate remaining subprocesses now.
{code}


upgrade_tests/upgrade_through_versions_test.py::TestProtoV3Upgrade_AllVersions_RandomPartitioner_EndsAt_3_11_X_HEAD:: test_rolling_upgrade_with_internode_ssl

{code}
>           raise RuntimeError(message)
E           RuntimeError: A subprocess has terminated early. Subprocess statuses: Process-1 (is_alive: True), Process-2 (is_alive: False), attempting to terminate remaining subprocesses now.
{code}


upgrade_tests/upgrade_through_versions_test.py::TestProtoV3Upgrade_AllVersions_RandomPartitioner_EndsAt_3_11_X_HEAD:: test_rolling_upgrade

{code}
>           raise RuntimeError(message)
E           RuntimeError: A subprocess has terminated early. Subprocess statuses: Process-1 (is_alive: True), Process-2 (is_alive: False), attempting to terminate remaining subprocesses now.
{code}


upgrade_tests/upgrade_through_versions_test.py::TestProtoV3Upgrade_AllVersions_EndsAt_3_11_X:: test_rolling_upgrade_with_internode_ssl

{code}
>           raise RuntimeError(message)
E           RuntimeError: A subprocess has terminated early. Subprocess statuses: Process-1 (is_alive: True), Process-2 (is_alive: False), attempting to terminate remaining subprocesses now.
{code}