Bug: Cannot run restore for a list of Cassandra nodes
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 25/Mar/21 04:20
Resolved: 25/Mar/21 14:32

Hi,

 

When switching to use Cassandra medus to fulfill our work for node data restore, we encountered some issues.

When using pssh remotely we are getting timeout issue, when trying the command on one node of Cassandra, we  get

 
{code:java}
pssh -H XXXX medusa -vvv restore-node --in-place --no-verify --backup-name 2021031803 --temp-dir /tmp/medusa-job-bd8a39ca-a5ea-4a3a-820f-0fa6ddc5130a
 [1] 06:52:08 [FAILURE] sha8392 Timed out, Killed by signal 9
 When further looking into the timeout issue, we get logs as
 [2021-03-25 02:23:50,113] DEBUG: https://s3.cn-north-1.amazonaws.com.cn:443 "GET /XX/XX/10.44.XX.XX/2021031803/meta/schema.cql?Version=2006-03-01 HTTP/1.1" 200 24005[2021-03-25 02:23:50,114] DEBUG: [Storage] Getting object sre_dev_cass_sha/10.44.79.15/2021031803/meta/tokenmap.json
 [2021-03-25 02:23:50,151] DEBUG: https://s3.cn-north-1.amazonaws.com.cn:443 "HEAD /XX HTTP/1.1" 200 0[2021-03-25 02:23:50,201] DEBUG: https://s3.cn-north-1.amazonaws.com.cn:443 "HEAD /XX/XX/10.44.79.15/2021031803/meta/tokenmap.json HTTP/1.1" 200 0[2021-03-25 02:23:50,202] DEBUG: Downloading /tmp/medusa-job-bd8a39ca-a5ea-4a3a-820f-0fa6ddc5130a/medusa-restore-197b6c82-4cd5-4c5b-b3c2-9d98863c1b3f as single part
 [2021-03-25 02:23:50,254] DEBUG: https://s3.cn-north-1.amazonaws.com.cn:443 "GET /XX/XX/10.44.XX.XX/2021031803/meta/tokenmap.json?Version=2006-03-01 HTTP/1.1" 200 1535[2021-03-25 02:23:50,255] INFO: Stopping Cassandra
+ /usr/bin/nodetool u cassandra -pw if9te8ohKei9xaep drain+ /usr/bin/nodetool -u cassandra -pw if9te8ohKei9xaep drainerror: null- StackTrace --java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:267) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298) at com.sun.proxy.$Proxy8.drain(Unknown Source) at org.apache.cassandra.tools.NodeProbe.drain(NodeProbe.java:371) at org.apache.cassandra.tools.nodetool.Drain.execute(Drain.java:36) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244) at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
 + ls -l /var/run/cassandra/cassandra.pidls: cannot access /var/run/cassandra/cassandra.pid: No such file or directory+ sleep 10+ echo -n 'Shutdown Cassandra: 'Shutdown Cassandra: ++ cat /var/run/cassandra/cassandra.pidcat: /var/run/cassandra/cassandra.pid: No such file or directory+ su cassandra -c 'kill 'kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]++ seq 40+ for t in '`seq 40`'+ /etc/init.d/cassandra status+ break+ sleep 5+ echo OKOK
{code}
But we can get a successful run of the command on one node for
{code:java}
export LC_ALL=en_US.UTF-8; export LANG=en_US.UTF-8; export https_proxy=http://proxy.XX:3128 ; export PATH=$PATH:/usr/share/cassandra-medusa/bin; sudo su; mkdir /tmp/medusa-job-bd8a39ca-a5ea-4a3a-820f-0fa6ddc5130a; cd /tmp/medusa-job-bd8a39ca-a5ea-4a3a-820f-0fa6ddc5130a;
medusa-wrapper sudo 
medusa -vvv restore-node --in-place --no-verify --backup-name 2021031803 --temp-dir /tmp/medusa-job-bd8a39ca-a5ea-4a3a-820f-0fa6ddc5130a{code}
We are running the command on 
{code:java}
uname -a
Linux XXXX 5.3.0-53-generic #47~18.04.1-Ubuntu SMP Thu May 7 13:10:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux{code}
Could you please have a look at the issue?

Thanks