Bug: Find and remove duplicate execution of dtests enabled by inheritance
Status: Triage Needed
Priority: Normal
Created: 13/Sep/21 12:05

Patch to CASSANDRA-16841 helped to reveal that there are tests, which are selected multiple times for execution. The discussion starts from [this comment|https://issues.apache.org/jira/browse/CASSANDRA-16841?focusedCommentId=17413128&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17413128]. The reason for double test execution is having a test class inheriting another test class, which means that test methods in the parent test class will be selected for the execution two times: one as part of their class, i.e., the parent class, and one as part of the child class, since it inherits them.

In particular I noticed two different cases:
 1. A base class is a test class according to PyTest convention and it contains test methods in it. It is imported in different dtest file, e.g., in upgrade folder to be executed as upgrade test, where it is inherited in a child class, which is also a test class. Just in the second file the test methods from the base class will be present two times: once due to import and once due to inheritance.
 2. A base test class is inherited inside the same file and it is assumed in the file that the tests need to be executed only once. However, the tests will be selected two times, since the base test class will be selected for execution, and the child test class will include the test methods from the parent test class and, thus, the tests will be selected for the execution the second time.

I illustrate this on a [commit before my fix|https://github.com/apache/cassandra-dtest/commit/efb82f574cd99ea598713035bbc9a95e568d73ce].

1. [upgrade_tests/bootstrap_upgrade_test.py|https://github.com/apache/cassandra-dtest/blob/efb82f574cd99ea598713035bbc9a95e568d73ce/upgrade_tests/bootstrap_upgrade_test.py] imports a test class {{TestBootstrap}} and then a test class {{TestBootstrapUpgrade}} inherits from it. This results in executing tests from {{TestBootstrap}} two times:
{code}
pytest --cassandra-dir=../cassandra/ upgrade_tests/bootstrap_upgrade_test.py --collect-only --execute-upgrade-tests-only
...
collected 57 items                                                                                                                                                                             
<Module 'upgrade_tests/bootstrap_upgrade_test.py'>
  <Class 'TestBootstrap'>
    <Instance '()'>
      <Function 'test_simple_bootstrap_with_ssl'>
      <Function 'test_simple_bootstrap'>
      <Function 'test_bootstrap_on_write_survey'>
      <Function 'test_simple_bootstrap_small_keepalive_period'>
      <Function 'test_simple_bootstrap_nodata'>
      <Function 'test_schema_removed_nodes'>
      <Function 'test_read_from_bootstrapped_node'>
      <Function 'test_bootstrap_waits_for_streaming_to_finish'>
      <Function 'test_consistent_range_movement_true_with_replica_down_should_fail'>
      <Function 'test_consistent_range_movement_false_with_replica_down_should_succeed'>
      <Function 'test_consistent_range_movement_true_with_rf1_should_fail'>
      <Function 'test_consistent_range_movement_false_with_rf1_should_succeed'>
      <Function 'test_rf_gt_nodes_multidc_should_succeed'>
      <Function 'test_resumable_bootstrap'>
      <Function 'test_bootstrap_with_reset_bootstrap_state'>
      <Function 'test_manual_bootstrap'>
      <Function 'test_local_quorum_bootstrap'>
      <Function 'test_shutdown_wiped_node_cannot_join'>
      <Function 'test_killed_wiped_node_cannot_join'>
      <Function 'test_decommissioned_wiped_node_can_join'>
      <Function 'test_decommissioned_wiped_node_can_gossip_to_single_seed'>
      <Function 'test_failed_bootstrap_wiped_node_can_join'>
      <Function 'test_node_cannot_join_as_hibernating_node_without_replace_address'>
      <Function 'test_simultaneous_bootstrap'>
      <Function 'test_cleanup'>
      <Function 'test_bootstrap_binary_disabled'>
      <Function 'test_invalid_host_id'>
      <Function 'test_host_id_override'>
  <Class 'TestBootstrapUpgrade'>
    <Instance '()'>
      <Function 'test_simple_bootstrap_with_ssl'>
      <Function 'test_simple_bootstrap'>
      <Function 'test_bootstrap_on_write_survey'>
      <Function 'test_simple_bootstrap_small_keepalive_period'>
      <Function 'test_simple_bootstrap_nodata'>
      <Function 'test_schema_removed_nodes'>
      <Function 'test_read_from_bootstrapped_node'>
      <Function 'test_bootstrap_waits_for_streaming_to_finish'>
      <Function 'test_consistent_range_movement_true_with_replica_down_should_fail'>
      <Function 'test_consistent_range_movement_false_with_replica_down_should_succeed'>
      <Function 'test_consistent_range_movement_true_with_rf1_should_fail'>
      <Function 'test_consistent_range_movement_false_with_rf1_should_succeed'>
      <Function 'test_rf_gt_nodes_multidc_should_succeed'>
      <Function 'test_resumable_bootstrap'>
      <Function 'test_bootstrap_with_reset_bootstrap_state'>
      <Function 'test_manual_bootstrap'>
      <Function 'test_local_quorum_bootstrap'>
      <Function 'test_shutdown_wiped_node_cannot_join'>
      <Function 'test_killed_wiped_node_cannot_join'>
      <Function 'test_decommissioned_wiped_node_can_join'>
      <Function 'test_decommissioned_wiped_node_can_gossip_to_single_seed'>
      <Function 'test_failed_bootstrap_wiped_node_can_join'>
      <Function 'test_node_cannot_join_as_hibernating_node_without_replace_address'>
      <Function 'test_simultaneous_bootstrap'>
      <Function 'test_cleanup'>
      <Function 'test_bootstrap_binary_disabled'>
      <Function 'test_invalid_host_id'>
      <Function 'test_host_id_override'>
      <Function 'test_simple_bootstrap_mixed_versions'>
{code}
2. [/sstable_generation_loading_test.py|https://github.com/apache/cassandra-dtest/blob/efb82f574cd99ea598713035bbc9a95e568d73ce/sstable_generation_loading_test.py] defines a base class {{TestBaseSStableLoader}}, which is inherited by {{TestSSTableGenerationAndLoading}} without doing any changes to test configurations. As result the test methods from the base class are selected for execution two times:
{code}
pytest --cassandra-dir=../cassandra/ sstable_generation_loading_test.py --collect-only                                  
...
collected 24 items                                                                                                                                                                             
<Module 'sstable_generation_loading_test.py'>
  <Class 'TestBaseSStableLoader'>
    <Instance '()'>
      <Function 'test_sstableloader_compression_none_to_none'>
      <Function 'test_sstableloader_compression_none_to_snappy'>
      <Function 'test_sstableloader_compression_none_to_deflate'>
      <Function 'test_sstableloader_compression_snappy_to_none'>
      <Function 'test_sstableloader_compression_snappy_to_snappy'>
      <Function 'test_sstableloader_compression_snappy_to_deflate'>
      <Function 'test_sstableloader_compression_deflate_to_none'>
      <Function 'test_sstableloader_compression_deflate_to_snappy'>
      <Function 'test_sstableloader_compression_deflate_to_deflate'>
      <Function 'test_sstableloader_with_mv'>
  <Class 'TestSSTableGenerationAndLoading'>
    <Instance '()'>
      <Function 'test_sstableloader_compression_none_to_none'>
      <Function 'test_sstableloader_compression_none_to_snappy'>
      <Function 'test_sstableloader_compression_none_to_deflate'>
      <Function 'test_sstableloader_compression_snappy_to_none'>
      <Function 'test_sstableloader_compression_snappy_to_snappy'>
      <Function 'test_sstableloader_compression_snappy_to_deflate'>
      <Function 'test_sstableloader_compression_deflate_to_none'>
      <Function 'test_sstableloader_compression_deflate_to_snappy'>
      <Function 'test_sstableloader_compression_deflate_to_deflate'>
      <Function 'test_sstableloader_uppercase_keyspace_name'>
      <Function 'test_incompressible_data_in_compressed_table'>
      <Function 'test_remove_index_file'>
      <Function 'test_sstableloader_with_mv'>
      <Function 'test_sstableloader_with_failing_2i'>
{code}

[This commit|https://github.com/apache/cassandra-dtest/pull/152/commits/4952c6392a4cc5735a6c8488312defab930ac543] fixes the above cases, however it is likely to be more of them in dtest repo. It is possible to discover them in two ways:
1. Run pytest with collect only and see if there are duplications. It will require to use additional options to enable marked tests.
2. Search for test classes inheriting other test classes, i.e., according the pattern {{class Test*(Test*)}}

Note that there will be false positives, since there are cases to inherit on purpose with different configuration. See [this example|https://github.com/apache/cassandra-dtest/blob/efb82f574cd99ea598713035bbc9a95e568d73ce/upgrade_tests/storage_engine_upgrade_test.py#L514]

It will be good to implement a way to prevent such issue in future.