hpcc-systems/HPCC-Platform — Issue #1185: backupnode not working
Author: cloLN
Created: 2012-01-05 19:04:54+00:00
State: open

according to @rwagner42 backupnode is not working

When attempting to run backupnode, it returns:

/opt/HPCCSystems/bin/start_backupnode /var/lib/HPCCSystems/thor
------------------------------
starting backupnode ...
/opt/HPCCSystems/bin/start_backupnode: line 56: cd: /var/lib/HPCCSystems//var/lib/HPCCSystems/thor: No such file or directory
cannot read /var/lib/HPCCSystems//var/lib/HPCCSystems/thor/slaves

To get past that error, I modified /opt/HPCCSystems/bin/start_backupnode line 56 from:

INSTANCE_DIR=$RUN_DIR/$1

To:

INSTANCE_DIR=$1

Re-running it then returned the following:

/opt/HPCCSystems/bin/start_backupnode /var/lib/HPCCSystems/thor
------------------------------
starting backupnode ...
------------------------------
scanning files from dali ...
/opt/HPCCSystems/bin/start_backupnode: line 94: syntax error near unexpected token `>'
/opt/HPCCSystems/bin/start_backupnode: line 94: `$DEPLOY_DIR/backupnode -O $DALISERVER $NODEGROUP $BACKUPNODE_DATA &>> $LOGFILE'

This error is due to us running bash 3.2.25, and the stderr/stdout redirect "&>>" is only supported in bash 4 final. So I changed the redirects on line 94 and 101 to use "2>&1 >>" instead.

Re-running backupnode then returned:

/opt/HPCCSystems/bin/start_backupnode /var/lib/HPCCSystems/thor
------------------------------
starting backupnode ...
------------------------------
scanning files from dali ...
00000000 2011-12-22 10:55:54 24750 24750 "Creating part lists, please wait..."
00000001 2011-12-22 10:56:22 24750 24750 "backupnode: File scan complete, 9202 files, 366571 parts"
00000002 2011-12-22 10:56:22 24750 24750 "backupnode finished"
ERROR: /var/jenkins/workspace/EE-Candidate-3.4.0/HPCC-Platform/services/runagent/frunssh.cpp(73) : frunssh : Cannot find identity file: /root/.ssh/id_rsa
ERROR: /var/jenkins/workspace/EE-Candidate-3.4.0/HPCC-Platform/services/runagent/frunssh.cpp(73) : frunssh : Cannot find identity file: /root/.ssh/id_rsa
------------------------------
------------------------------
Waiting for backup to complete

I stopped the backupnode process, created root keys for all nodes and distributed them. Re-running it then showed:

/opt/HPCCSystems/bin/start_backupnode /var/lib/HPCCSystems/thor
------------------------------
starting backupnode ...
------------------------------
scanning files from dali ...
00000000 2011-12-22 11:02:03 25078 25078 "Creating part lists, please wait..."
00000001 2011-12-22 11:02:31 25078 25078 "backupnode: File scan complete, 9202 files, 366571 parts"
00000002 2011-12-22 11:02:31 25078 25078 "backupnode finished"
33: starting 10.239.218.34 (0 of 40 finished)
18: starting 10.239.218.19 (0 of 40 finished)
29: starting 10.239.218.30 (0 of 40 finished)
13: starting 10.239.218.14 (0 of 40 finished)
38: starting 10.239.218.39 (0 of 40 finished)
14: starting 10.239.218.15 (0 of 40 finished)
8: starting 10.239.218.9 (0 of 40 finished)
4: starting 10.239.218.5 (0 of 40 finished)
26: starting 10.239.218.27 (0 of 40 finished)
5: starting 10.239.218.6 (0 of 40 finished)
9: starting 10.239.218.10 (1 of 40 finished)
3: starting 10.239.218.4 (2 of 40 finished)
30: starting 10.239.218.31 (3 of 40 finished)
7: starting 10.239.218.8 (4 of 40 finished)
37: starting 10.239.218.38 (5 of 40 finished)
27: starting 10.239.218.28 (6 of 40 finished)
17: starting 10.239.218.18 (7 of 40 finished)
10: starting 10.239.218.11 (8 of 40 finished)
39: starting 10.239.218.40 (9 of 40 finished)
25: starting 10.239.218.26 (10 of 40 finished)
34: starting 10.239.218.35 (11 of 40 finished)
23: starting 10.239.218.24 (12 of 40 finished)
2: starting 10.239.218.3 (13 of 40 finished)
19: starting 10.239.218.20 (14 of 40 finished)
22: starting 10.239.218.23 (15 of 40 finished)
12: starting 10.239.218.13 (16 of 40 finished)
0: starting 10.239.218.1 (17 of 40 finished)
16: starting 10.239.218.17 (18 of 40 finished)
28: starting 10.239.218.29 (19 of 40 finished)
31: starting 10.239.218.32 (20 of 40 finished)
1: starting 10.239.218.2 (21 of 40 finished)
6: starting 10.239.218.7 (22 of 40 finished)
21: starting 10.239.218.22 (23 of 40 finished)
35: starting 10.239.218.36 (24 of 40 finished)
15: starting 10.239.218.16 (25 of 40 finished)
36: starting 10.239.218.37 (26 of 40 finished)
32: starting 10.239.218.33 (27 of 40 finished)
20: starting 10.239.218.21 (28 of 40 finished)
24: starting 10.239.218.25 (29 of 40 finished)
11: starting 10.239.218.12 (30 of 40 finished)
Results: (40 of 40 finished)
34: 10.239.218.34(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
19: 10.239.218.19(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
30: 10.239.218.30(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
14: 10.239.218.14(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
39: 10.239.218.39(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
15: 10.239.218.15(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
9: 10.239.218.9(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
5: 10.239.218.5(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
27: 10.239.218.27(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
6: 10.239.218.6(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
10: 10.239.218.10(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
4: 10.239.218.4(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
31: 10.239.218.31(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
8: 10.239.218.8(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
38: 10.239.218.38(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
28: 10.239.218.28(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
18: 10.239.218.18(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
11: 10.239.218.11(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
40: 10.239.218.40(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
26: 10.239.218.26(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
35: 10.239.218.35(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
24: 10.239.218.24(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
3: 10.239.218.3(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
20: 10.239.218.20(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
23: 10.239.218.23(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
13: 10.239.218.13(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
1: 10.239.218.1(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
17: 10.239.218.17(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
29: 10.239.218.29(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
32: 10.239.218.32(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
2: 10.239.218.2(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
7: 10.239.218.7(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
22: 10.239.218.22(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
36: 10.239.218.36(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
16: 10.239.218.16(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
37: 10.239.218.37(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
33: 10.239.218.33(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
21: 10.239.218.21(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
25: 10.239.218.25(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
12: 10.239.218.12(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
39: starting 10.239.218.40 (0 of 40 finished)
34: starting 10.239.218.35 (0 of 40 finished)
6: starting 10.239.218.7 (0 of 40 finished)
11: starting 10.239.218.12 (0 of 40 finished)
20: starting 10.239.218.21 (0 of 40 finished)
37: starting 10.239.218.38 (0 of 40 finished)
21: starting 10.239.218.22 (0 of 40 finished)
29: starting 10.239.218.30 (0 of 40 finished)
38: starting 10.239.218.39 (0 of 40 finished)
7: starting 10.239.218.8 (0 of 40 finished)
12: starting 10.239.218.13 (1 of 40 finished)
8: starting 10.239.218.9 (2 of 40 finished)
3: starting 10.239.218.4 (3 of 40 finished)
26: starting 10.239.218.27 (4 of 40 finished)
30: starting 10.239.218.31 (5 of 40 finished)
10: starting 10.239.218.11 (6 of 40 finished)
4: starting 10.239.218.5 (7 of 40 finished)
19: starting 10.239.218.20 (8 of 40 finished)
5: starting 10.239.218.6 (9 of 40 finished)
28: starting 10.239.218.29 (10 of 40 finished)
16: starting 10.239.218.17 (11 of 40 finished)
31: starting 10.239.218.32 (12 of 40 finished)
32: starting 10.239.218.33 (13 of 40 finished)
2: starting 10.239.218.3 (14 of 40 finished)
1: starting 10.239.218.2 (15 of 40 finished)
9: starting 10.239.218.10 (16 of 40 finished)
0: starting 10.239.218.1 (17 of 40 finished)
25: starting 10.239.218.26 (18 of 40 finished)
36: starting 10.239.218.37 (19 of 40 finished)
14: starting 10.239.218.15 (20 of 40 finished)
13: starting 10.239.218.14 (21 of 40 finished)
17: starting 10.239.218.18 (22 of 40 finished)
35: starting 10.239.218.36 (23 of 40 finished)
18: starting 10.239.218.19 (24 of 40 finished)
33: starting 10.239.218.34 (25 of 40 finished)
27: starting 10.239.218.28 (26 of 40 finished)
22: starting 10.239.218.23 (27 of 40 finished)
15: starting 10.239.218.16 (28 of 40 finished)
24: starting 10.239.218.25 (29 of 40 finished)
23: starting 10.239.218.24 (30 of 40 finished)
Results: (40 of 40 finished)
40: 10.239.218.40(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
35: 10.239.218.35(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
7: 10.239.218.7(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
12: 10.239.218.12(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
21: 10.239.218.21(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
38: 10.239.218.38(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
22: 10.239.218.22(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
30: 10.239.218.30(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
39: 10.239.218.39(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
8: 10.239.218.8(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
13: 10.239.218.13(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
9: 10.239.218.9(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
4: 10.239.218.4(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
27: 10.239.218.27(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
31: 10.239.218.31(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
11: 10.239.218.11(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
5: 10.239.218.5(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
20: 10.239.218.20(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
6: 10.239.218.6(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
29: 10.239.218.29(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
17: 10.239.218.17(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
32: 10.239.218.32(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
33: 10.239.218.33(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
3: 10.239.218.3(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
2: 10.239.218.2(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
10: 10.239.218.10(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
1: 10.239.218.1(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
26: 10.239.218.26(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
37: 10.239.218.37(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
15: 10.239.218.15(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
14: 10.239.218.14(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
18: 10.239.218.18(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
36: 10.239.218.36(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
19: 10.239.218.19(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
34: 10.239.218.34(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
28: 10.239.218.28(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
23: 10.239.218.23(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
16: 10.239.218.16(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
25: 10.239.218.25(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
24: 10.239.218.24(255): ERR: Permission denied (publickey,gssapi-with-mic,password).
------------------------------
------------------------------
Waiting for backup to complete

I noted that it says (0 of 40 finished) 10 times before correctly incrementing to (1 of 40 finished) and finishes at (30 of 40 finished). I've not been able to figure out what is causing that, nor what is causing the Permission denied's. I've confirmed that the root key is working as expected and I can ssh to all nodes as root without a password from the thormaster node.

Please advise.

@pschwartz @jakesmith @fuceta