av9ash/GitBugs huggingface dataset:
Bug: Run Jenkins Cqlsh Tests using the in-tree cassandra-cqlsh-tests.sh
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 06/Jan/20 12:12
Resolved: 07/Jan/20 09:05

Make the Jenkins Cqlsh test job to use the in-tree cassandra-cqlsh-tests.sh instead of the same script from the cassandra-builds repo.
Bug: Empty clustering instead of Clustering.EMPTY
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 10/Jan/20 17:40
Resolved: 10/Jan/20 18:08

It seems that there are several ways to produce an empty clustering (with Kind.CLUSTERING) without using Clustering.EMPTY. They can be produced when copying another empty clustering (such as Clustering.EMPTY) , and also during deserialization when calling to Clustering.make
Bug: cqlsh COPY FROM of map of blobs fails with parse error "unhashable type: 'bytearray'"
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 31/Mar/20 01:16
Resolved: 09/Apr/20 13:31

h2. Background
A user was having issues loading CSV data with the {{COPY FROM}} command into a {{map}} column with {{blob}} values.
h2. Replication steps
I can easily replicate the problem with this simple table:

{noformat}
CREATE TABLE community.blobmaptable (
    id text PRIMARY KEY,
    blobmapcol map<int, blob>
)
{noformat}

I have this CSV file that contains just 1 row:

{noformat}
$ cat blobmap.csv 
c3,{3: 0x74776f}
{noformat}

And here's the error when I try to load it:

{noformat}
cqlsh:community> COPY blobmaptable (id, blobmapcol) FROM '~/blobmap.csv' ;
Using 1 child processes
Starting copy of community.blobmaptable with columns [id, blobmapcol].
Failed to import 1 rows: ParseError - Failed to parse {3: 0x74776f} : unhashable type: 'bytearray',  given up without retries
Failed to process 1 rows; failed rows written to import_community_blobmaptable.err
Processed: 1 rows; Rate:       2 rows/s; Avg. rate:       3 rows/s
1 rows imported from 1 files in 0.389 seconds (0 skipped).
{noformat}

I've also logged [PYTHON-1234|https://datastax-oss.atlassian.net/browse/PYTHON-1234] because I wasn't sure if it was a Python driver issue. Cheers!
Bug: Documentation gives the wrong instruction to activate remote jmx
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 30/Jan/20 15:14
Resolved: 13/Feb/20 22:13

In this section [jmx acces |https://cassandra.apache.org/doc/latest/operating/security.html?highlight=local_jmx#jmx-access], in order to activate the remove jmx acces, it says to change the value of LOCAL_JMX to *yes*. However the right configuration is *no*
Bug: Better support of Python 3 for cqlsh
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Mar/20 19:04
Resolved: 14/Apr/20 19:02

h2. From mailing list:

[https://lists.apache.org/thread.html/r377099b632c62b641e4feef5b738084fc5369b0c7157fae867853597%40%3Cdev.cassandra.apache.org%3E]

 

As of today (24/3/2020) and current trunk, there is Python 3.6 supported (1) but there is not any 3.6 version ootb in Debian for example. E.g. Buster has Python 3.7 and other (recent) releases have version 2.7. This means that if one wants to use Python 3 in Debian, he has to use 3.6 but it is not in the repository so he has to download / compile / install it on his own.

There should be some sane Python 3 version supported which is as well present in Debian repository (or requirement to run with 3.6 should be relaxed) .

(1) [https://github.com/apache/cassandra/blob/bf9a1d487b9ba469e8d740cf7d1cd419535a7e79/bin/cqlsh#L57-L65]
h2. Summary of work that was done:

I relaxed the requirement of *cqlsh* only working with Python 2.7 & 3.6 by allowing Python 3.6+.
 Note that I left the constraint for Python 3.6 being the minimum Python3 version. 
 As [~ptbannister] pointed out, we could remove the Python 3.6 min version once we remove Python 2.7 support, as otherwise testing with lots of different Python versions will get costly.

2 Dockerfiles were added in *pylib* for minimal local testing of *cqlsh* starting up with Python 3.7 & 3.8 and that both revealed
 CASSANDRA-15572 and CASSANDRA-15573. 
 CASSANDRA-15572 was fixed here as it was a one-liner. And I'm going to tackle CASSANDRA-15573 later.

Python 3.8 testing was added to the CircleCI config so that we can actually see what else breaks with newer Python versions.

A new Docker images with Ubuntu 19.10 was required for testing ([https://github.com/apache/cassandra-builds/pull/17]). This docker image sets up Python 2.7/3.6/3.7/3.8 with their respective virtual environments, which are then being used by the CircleCI yaml.

The image *spod/cassandra-testing-ubuntu1810-java11-w-dependencies:20190306* couldn't be updated unfortunately because it can't be built anymore, due to Ubuntu 18.10 being EOL.
Bug: Unit tests throw UnknownHostException trying to use `InetAddress.getLocalHost()` instead of `FBUtilities.getLocalAddress()`
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Mar/20 19:03
Resolved: 11/Mar/20 16:44

Many of the unit tests in Jenkins fail because of the use of {{`InetAddress.getLocalHost()`}} in the test classes.

The Jenkins agents need a public ipaddress (and a hostname associated to it) so the Jenkins master can connect to them (agents can be hosted externally, by donating third-parties).

The call to {{`InetAddress.getLocalHost()`}} can resolve to a hostname that can't be looked up.
Not only can it not be listed in {{`/etc/hosts`}}, but we don't want it to be either (in case of accidental external port opening if the hostname points to the public ipaddress). (Which is also ASF policy on this infrastructure.)

The unit test code needs to replace these code occurrences with the call to {{`FBUtilities.getLocalAddress()`}}
Bug: Optionally set nowInSec in the future for validation repairs
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Mar/20 09:41
Resolved: 03/Apr/20 10:57

There is a race when running validation repairs where we might build merkle trees on one node, have gcgs expire, tombstones get compacted out and then build merkle trees on other nodes, causing false positive preview repair mismatch
Bug: python dtest function assert_stderr_clean should exclude JAVA_TOOL_OPTIONS
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 29/Jan/20 21:24
Resolved: 31/Jan/20 18:14

JAVA_TOOL_OPTIONS is a JVM override to allow setting JVM flags without modifying all code which creates the JVM, but for security reasons (see [https://bugs.openjdk.java.net/browse/JDK-8039152]) it needs to log to stderr. Python dtest uses the function assert_stderr_clean to check to see if stderr had any unexpected errors, and should whitelist JAVA_TOOL_OPTIONS so it doesn’t cause test failures.

PR: https://github.com/apache/cassandra-dtest/pull/56

Testing done
{code}
export JAVA_TOOL_OPTIONS='-ea'
pytest \
            --log-level="DEBUG" \
            --log-cli-level="DEBUG" \
            --junit-xml=pytest_result.xml \
            --cassandra-dir="/cassandra" \
            --keep-test-dir \
            bootstrap_test::TestBootstrap::test_local_quorum_bootstrap
{code}
Bug: liveDiskSpaceUsed and totalDiskSpaceUsed get corrupted if IndexSummaryRedistribution gets interrupted
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Mar/20 23:50
Resolved: 07/May/20 10:28

IndexSummaryRedistribution is a compaction task and as such extends Holder and supports cancelation by throwing a CompactionInterruptedException.  The issue is that IndexSummaryRedistribution tries to use transactions, but mutates the sstable in-place; transaction is unable to roll back.

This would be fine (only updates summary) if it wasn’t for the fact the task attempts to also mutate the two metrics liveDiskSpaceUsed and totalDiskSpaceUsed, since these can’t be rolled back any cancelation could corrupt these metrics.
Bug:  Improve logging by using more appropriate levels
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Mar/20 20:35
Resolved: 01/Apr/20 15:19

There are a number of log statements using logging levels that are a bit too conservative.  For example:

* Flushing memtables is currently at debug.  This is a relatively rare event that is important enough to be INFO
* When compaction finishes we log the progress at debug
* Different steps in incremental repair are logged as debug, should be INFO
* when reaching connection limits in ConnectionLimitHandler.java we log at warn rather than error.  Since this is a client disconnect it’s more than a warning, we’re taking action and disconnecting.
Bug: No error if consistency_level = SERIAL and unqualified select
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 13/Mar/20 17:19
Resolved: 02/Apr/20 09:02

The attached test program produces no errors, while doesn't provide serial consistency. There is no exception that the consistency level is incorrect. It's simply silently downgraded. The issue is not documented either. 

kostja@atlas ~ % python test.py
Row(a=1, b=1)
Row(a=0, b=0)
Row(a=2, b=2)
Row(a=3, b=3)
Row(a=1, b=1)
Row(a=2, b=2)

The behavior is contrary to the original intent by LWT author, since the code has the following check: 

   if (group.queries.size() > 1)
            throw new InvalidRequestException("SERIAL/LOCAL_SERIAL consistency may only be requested for one partition at a time");
https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/service/StorageProxy.java#L1593
Bug: Fix flakey testSendSmall - org.apache.cassandra.net.ConnectionTest
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 10/Mar/20 16:40
Resolved: 26/Mar/20 09:29

The test fails sometimes with the following error message and trace. 

{code:java}
processed count values don't match expected:<10> but was:<9>

junit.framework.AssertionFailedError: processed count values don't match expected:<10> but was:<9>
at org.apache.cassandra.net.ConnectionUtils$InboundCountChecker.doCheck(ConnectionUtils.java:217)
at org.apache.cassandra.net.ConnectionUtils$InboundCountChecker.check(ConnectionUtils.java:200)
at org.apache.cassandra.net.ConnectionTest.lambda$testSendSmall$11(ConnectionTest.java:305)
at org.apache.cassandra.net.ConnectionTest.lambda$doTest$8(ConnectionTest.java:242)
at org.apache.cassandra.net.ConnectionTest.doTestManual(ConnectionTest.java:262)
at org.apache.cassandra.net.ConnectionTest.doTest(ConnectionTest.java:240)
at org.apache.cassandra.net.ConnectionTest.test(ConnectionTest.java:229)
at org.apache.cassandra.net.ConnectionTest.testSendSmall(ConnectionTest.java:277)
{code}
Bug: Jenkins 'Cassandra' label applied to the declarative pipeline 
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Mar/20 11:52
Resolved: 28/Mar/20 10:57

On the new ci-cassandra.apache.org infrastructure agents are siloed to projects.

The declarative pipeline in the 2.2, 3.0, 3.11, and trunk branches do not restrict the builds to agents with the 'cassandra' label. Because these agents will not run jobs that don't specify this label, the pipeline task only ever runs on unlabelled agents, of which there are very few (and likely shouldn't exist, existing only from for now because of misconfigurations).

Example of the failure to run the pipeline tasks is
{noformat}
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
'cassandra10' is reserved for jobs with matching label expression; 'cassandra11' is reserved for jobs with matching label expression; 'cassandra12' is reserved for jobs with matching label expression; 'cassandra13' is reserved for jobs with matching label expression; 'cassandra14' is reserved for jobs with matching label expression; 'cassandra15' is reserved for jobs with matching label expression; 'cassandra16' is reserved for jobs with matching label expression; 'cassandra1' is reserved for jobs with matching label expression; 'cassandra2' is reserved for jobs with matching label expression; 'cassandra3' is reserved for jobs with matching label expression; 'cassandra4' is reserved for jobs with matching label expression; 'cassandra5' is reserved for jobs with matching label expression; 'cassandra6' is reserved for jobs with matching label expression; 'cassandra7' is reserved for jobs with matching label expression; 'cassandra8' is reserved for jobs with matching label expression; 'cassandra9' is reserved for jobs with matching label expression
{noformat}

Along with this change, we can improve the name of the {{*-test-jvm-dtest-forking}} stages.
Bug: Jenkins build for jvm test should use testclasslist to support parameterized tests
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Mar/20 03:05
Resolved: 15/Mar/20 22:37

We switched Circle CI to use testclasslist in CASSANDRA-15508 this was to solve the following exception

{code}
ava.lang.Exception: No tests found matching Method testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest) from org.junit.internal.requests.ClassRequest@551aa95a
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
{code}

The core issue is that the test locating logic in org.apache.cassandra.distributed.test.TestLocator does not handle parameterized tests so fails to find any tests for those classes.

I think it is better to switch to testclasslist as it helps make way for running tests concurrently.
Bug: Add dch release version to debian packages
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 04/Feb/20 06:41
Resolved: 24/Apr/20 13:21

Add to the prepare_release.sh script the `dch -r ` step that adds the version.

Currently the script will generate an UNRELEASED version.

Reference: https://lists.apache.org/thread.html/r7f8addef88553955b1057a61b29526cb974cc9037f20ce46fb26fb61%40%3Cdev.cassandra.apache.org%3E
Bug: Python 3.8 fails to execute cqlsh
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Feb/20 23:51
Resolved: 14/Apr/20 19:14

Python 3.8 renamed sre_parse.Pattern to sre_parse.State (see [https://bugs.python.org/issue34681] and corresponding pull request [https://github.com/python/cpython/pull/9310])

So when executing cqlsh with Python 3.8, it throws error:
{code:java}
Traceback (most recent call last):
  File ".\bin\cqlsh.py", line 175, in <module>
    from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling, cqlshhandling
  File "C:\Users\Yuki Morishita\Projects\cassandra\bin\..\pylib\cqlshlib\cql3handling.py", line 19, in <module>
    from cqlshlib.cqlhandling import CqlParsingRuleSet, Hint
  File "C:\Users\Yuki Morishita\Projects\cassandra\bin\..\pylib\cqlshlib\cqlhandling.py", line 23, in <module>
    from cqlshlib import pylexotron, util
  File "C:\Users\Yuki Morishita\Projects\cassandra\bin\..\pylib\cqlshlib\pylexotron.py", line 342, in <module>
    class ParsingRuleSet:
  File "C:\Users\Yuki Morishita\Projects\cassandra\bin\..\pylib\cqlshlib\pylexotron.py", line 343, in ParsingRuleSet
    RuleSpecScanner = SaferScanner([
  File "C:\Users\Yuki Morishita\Projects\cassandra\bin\..\pylib\cqlshlib\saferscanner.py", line 74, in __init__
    s = re.sre_parse.Pattern()
AttributeError: module 'sre_parse' has no attribute 'Pattern'
{code}
h2. Summary of Work that was done

Added a Python 3.8 compatible SaferScanner implementation ([diff here|https://github.com/apache/cassandra/pull/518/commits/2e6813f0ef5817e5d8d655052d61ce75a5fc062c]). Note that the changes from CASSANDRA-15659 are required in order to verify that the issue is fixed.
Bug: Fix flaky test org.apache.cassandra.streaming.StreamTransferTaskTest testFailSessionDuringTransferShouldNotReleaseReferences
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 05/Feb/20 18:59
Resolved: 11/Mar/20 15:51

{code}
junit.framework.AssertionFailedError: expected:<0> but was:<1>
	at org.apache.cassandra.streaming.StreamTransferTaskTest.testFailSessionDuringTransferShouldNotReleaseReferences(StreamTransferTaskTest.java:181)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}

Failure was on java 11
Bug: Failing jvm dtest: FailingRepairTest.testFailingMessage
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Jan/20 17:33
Resolved: 29/Jan/20 08:47

It seems we can't run parameterized unit tests with {{ant testsome}}:

{code}
$ ant testsome -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest -Dtest.methods=testFailingMessage
....
[junit-timeout] Testcase: initializationError(org.junit.runner.manipulation.Filter):    Caused an ERROR
[junit-timeout] No tests found matching Method testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest) from org.junit.internal.requests.ClassRequest@4d95d2a2
[junit-timeout] java.lang.Exception: No tests found matching Method testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest) from org.junit.internal.requests.ClassRequest@4d95d2a2
[junit-timeout]         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
{code}
Bug: test.distributed.timeout no longer respected in CircleCI
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 20/Mar/20 20:51
Resolved: 27/Mar/20 11:31

After switching jvm dtest over testclasslist (CASSANDRA-15508) we no longer respect the dtest timeout and instead use the unit test timeout (4m vs 6m).

This does not impact Jenkins as I made sure to check that before calling testclasslist; though this does impact 2.2, 3.0, and 3.11 as well.

||Config||Trunk||3.11||3.0||2.2||
|LOWER||[Circle CI|https://circleci.com/workflow-run/04f7fbe2-1919-4da0-bf72-ba41b41c3072]| [Circle CI|https://circleci.com/workflow-run/69c64062-80f5-4d08-8ab0-dbe88ce7b236] | [Circle CI|https://circleci.com/workflow-run/499d2fbe-a8c1-4a27-9430-a9ebb40aad53] | [Circle CI|https://circleci.com/workflow-run/fedbd5b8-683d-4c59-a9f6-3aad5a6ba41d] |
|HIGHER| [Circle CI|https://circleci.com/workflow-run/036bbad1-541a-49dc-a567-cef2300fa847] | TBD | TBD | TBD |

CI Failures were flaky tests, below are their links
* CASSANDRA-15630
Bug: Allow EXTRA_CLASSPATH to work in tarball/source installations
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 12/Feb/20 16:16
Resolved: 26/Feb/20 17:52

Both the debian and redhat packaging modify cassandra.in.sh to support the EXTRA_CLASSPATH variable.  It would be nice to just put this in cassandra.in.sh proper so people without a package installation can use it as well.
Bug: Update CHANGES.txt with Add cross-DC latency metrics (CASSANDRA-11569)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 28/Jan/20 09:01
Resolved: 28/Jan/20 17:53

h1. Background
Question posed on ASF Slack on whether there is a metric for cross-DC latency. I then discovered that there's a typo in {{CHANGES.txt}}.

h1. Action
Replace:

{noformat}
 * Add cross-DC latency metrics (CASSANDRA-11596)
{noformat}

with:

{noformat}
 * Add cross-DC latency metrics (CASSANDRA-11569)
{noformat}
Bug: Jenkins tests to use testclasslist where possible (like CircleCI)
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 21/Mar/20 11:19
Resolved: 28/Mar/20 10:00

Following up on CASSANDRA-15639
 make all the jenkins test jobs run in the same manner.

This standards the approach across test jobs and to CircleCI, and will make it easier to parallelise test runs later on.
Bug: Unable to specify -e/--execute flag in cqlsh
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/Mar/20 19:07
Resolved: 16/Apr/20 16:54

From mailing list:

[https://lists.apache.org/thread.html/r377099b632c62b641e4feef5b738084fc5369b0c7157fae867853597%40%3Cdev.cassandra.apache.org%3E]

The bug looks like this:
{code:java}
$ /usr/bin/cqlsh -e 'describe keyspaces' -u cassandra -p cassandra 127.0.0.1

Usage: cqlsh.py [options] [host [port]]cqlsh.py: error: '127.0.0.1' is not a valid port number.
{code}
This is working in 3.x releases just fine but fails on 4.

The workaround for 4.x code as of today is to put these statements into file and use "-f" flag.
Bug: Broken dtest replication_test.py::TestSnitchConfigurationUpdate
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 27/Feb/20 11:10
Resolved: 14/Mar/20 12:30

Noticed this failing on a couple of CI runs and repros when running trunk locally and on CircleCI

2 or 3 tests are consistently failing:
 * {{test_rf_expand_gossiping_property_file_snitch}}
 * {{test_rf_expand_property_file_snitch}}
 * {{test_move_forwards_between_and_cleanup}}

[https://circleci.com/workflow-run/f23f13a9-bbdc-4764-8336-109517e137f1]
Bug: Duplicate results with DISTINCT queries in mixed mode 2.1/3.0
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 13/Jan/20 08:03
Resolved: 16/Apr/20 11:02

When a client switches coordinator from a 2.1 node to a 3.0 node it sends a 2.1 paging state to the 3.0 node. The 2.1 {{PagingState}} does not have {{remainingInPartition}} so on the 3.0 side we default this to Integer.MAX_VALUE. This value is then used to decide if the lastKey should be included in the result.
Bug: add docs for production recommendations
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 02/Mar/20 21:55
Resolved: 30/Mar/20 21:11

We have some inline comments in our yaml, but I think it would be helpful if we had a more lengthy section in the docs detailing the settings people should be using in production.  Some of these are unrelated to the yaml itself (like reducing read ahead), and putting them in the YAML doesn’t make much sense.  There are also other settings (like compression chunk length) that don’t fit there.  Let’s add a page under getting started called Production Recommendations.
Bug: Improve zero-copy-streaming containment check by using file sections
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 23/Mar/20 15:12
Resolved: 18/Apr/20 18:59

Currently zero copy streaming is only enabled for leveled-compaction strategy and it checks if all keys in the sstables are included in the transferred ranges.

This is very inefficient. The containment check can be improved by checking if transferred sections (the transferred file positions) cover entire sstable.

I also enabled ZCS for all compaction strategies since the new containment check is very fast..
Bug: `object of type 'NoneType' has no len()` error in cqlsh with python 3
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 16/Feb/20 22:51
Resolved: 07/Apr/20 23:26

Looks like in Python 3, webbrowser._tryorder can be NoneType and throw error like below:

{code}
Traceback (most recent call last):
  File ".\bin\cqlsh.py", line 99, in <module>
    if len(webbrowser._tryorder) == 0:
TypeError: object of type 'NoneType' has no len()
{code}
Bug: Test org.apache.cassandra.distributed.test.DistributedReadWritePathTest#failingReadRepairTest does not test a failing read repair and should be updated to actually trigger a failed read repair
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 15/Jan/20 21:52
Resolved: 17/Jan/20 13:50

The test org.apache.cassandra.distributed.test.DistributedReadWritePathTest#failingReadRepairTest makes a few assumptions which are not valid at the moment.

1) the write to node 1 and 2 have the same digest (they don’t, this is caused by the timestamp being different)
2) node 3 will participate with the read; it won’t give the fact that org.apache.cassandra.locator.ReplicaPlans#contactForRead will speculate the first 2 nodes always, so node 3 won’t get involved with the repair
3) node 3 will attempt to get repaired (it won’t because its never looked at)