1 | Page

Navy Cash Technical Support 

Troubleshooting Guide

Cluster Servers Restart Instructions

One of the most effective methods to get Windows systems back on line is a power cycle. This is very efficient

in getting things back to a correct state and we ask that you do this before requesting a remote session. We 

find reboots can be up to 80% successful in resolving issues.

Note – If you are waiting on an EOM batch to complete, reboots will not speed up or correct this process. 

Please be aware it can take Shore a few hours to fully process an EOM batch.

Instructions for 1471/3 Navy Cash Versions

The following process involves shutting down just the two Navy Cash Servers while leaving the Storage Array 

online (Snap Server). 

1) Login to Node 1

a. If you are unable to login to Node 1 using any of the 3 server accounts that you have a password for,

perform a hard shutdown by pressing the power button in for that server until it turns off

2) Once logged into Node 1 perform the following:

a. Click the Start Button

b. Select Shutdown

c. If prompted for a reason, enterreason (i.e. power loss upcoming/proactive power cycle).

3) Login to Node 2

a. If you are unable to login to Node 2 using any of the 3 server accounts that you have a password for,

perform a hard shutdown by pressing the power button in for that server until it turnsoff

4) Once logged into Node 2 perform thefollowing:

a. Press the Start Button

b. Select Shutdown

c. If prompted for a reason, enterreason (i.e. power loss upcoming/proactive power cycle).

5) Once both Nodes have shutdown, turn on ONLY Node1.

a. Note – If Snap Server is down, start it up and wait a few minutes before bringing Node 1 up.

6) Once Node 1 is back up, attempt to login to it as normal and then turn on Node2.

7) Once Node 2 is back up, attempt to login to it asnormal.

8) If you are unable to login to one or both nodes, contact CSU to have a case opened.

9) If you are able to login to at least 1 node, check the Failover Cluster Manager to see if all the services are showing as

online. If services are still showing as Offline and/or Failed, proceed to Failover Cluster Issues for All Navy Cash Versions

below.
2 | Page

Instructions for 1474 Navy Cash Versions

1474 Shutdown instructions:

1. From Physical Server 1 (MS1)

2. Open Failover Cluster Manager

a. Right click on ncsvr-cl.ncsvr-d.navycash.navy.mil

b. More Actions

c. Shut Down Cluster

3. Open Internet Explorer

a. In address bar type 10.10.10.10 and press enter

b. Login to SNAP Server

c. Maintenance

d. Shutdown/Restart

e. Shutdown

f. Shutdown

4. Open Hyper-V Manager

a. Select DC 1

b. Right click

c. Connect

d. Login

e. Shutdown

5. From Physical Server 2 (MS2)

a. Open Hyper-V Manager

b. Select DC 2

c. Right click

d. Connect

e. Login

f. Shutdown

Note if MS 1 is unavailable, do steps 1 and 2 on MS2 using 10.10.11.10 in IE.

You are now clear to shutdown both servers from the windows menu, if prompted for a reason enter current actions (i.e. 

preparing for power outage, recycling servers, etc.).

1474 Power up Instructions:

1. Power up Snap Server by tapping power button on front for no more than 1 second

a. Wait a few minutes for the Snap Server to fully come up before proceeding

2. Power on Physical Server 1 by tapping power button on front

a. Note – Powering on both servers at the same time will cause issues with Navy Cash as both servers try to take 

control of the Navy Cash Cluster, only power on Server 1.

b. Login

c. Open Hyper-V Manager, if virtual server is already running skip to step 3

d. Select DC1

e. Right click

f. Start

3. Power on Physical Server 2 by tapping button on front

a. Login

b. Open Hyper-V Manager, if virtual server is already running, skip to step 4

c. Select DC2

d. Right click

e. Start

4. Once everything is powered up completely open Failover Cluster Manager

a. Check if Cluster came up, otherwise proceed

b. Right click on ncsvr-cl.ncsvr-d.navycash.navy.mil

c. More Actions

d. Start Cluster
3 | Page

Port 22 Check for All Navy Cash Versions

On the Navy Cash Server perform the following steps to verify that port 22 is open.

1) Open a command prompt

2) At the ‘C:\>’ prompt, enter the following line:

a. nslookup gppfts.navycash.fiscal.treasury.gov

3) You should get a response that indicates 199.169.193.37

4) At the ‘C:\>’ prompt, enter the following line:

a. telnet gppfts.navycash.fiscal.treasury.gov22

5) If you get a response back that looks similar to the following then port 22 is open for the Navy cash Server:

6) If you get don’t get a response back that looks similar to one above then port 22 is not open for the Navy cash Server.

7) Repeat the process on the other Navy CashServer.

8) If you don’t see the results shown in Step 5 on either node, then you will want to get with the ship ITs to validate the

ACL is good on the ship for port 22.

9) If the ACL is good, then the ship ITs will need to open a BCR/FSR with the NOC that the ship is connecting to that

includes the following information:

a. Source IP address – External IP addresses for both Node 1 and Node2

b. Destination IP address – 199.169.193.37

c. Port – 22

d. Protocol – SSH

e. Traffic Direction – Two-way (bi-directional)

ACL Check for All Navy Cash Versions

If round trips have not been going through and you are unable to browse the Internet from the Navy Cash server, then the 

issue may be with the ship/barge ACL entries for the Navy Cash system.

Please have your ITs verify the following entries are in present:

access-list 110 permit tcp host <navy-cash node #1 IP> any eq 20

access-list 110 permit tcp host <navy-cash node #1 IP> any eq 21

access-list 110 permit tcp host <navy-cash node #1 IP> any eq 22

access-list 110 permit tcp host <navy-cash node #2 IP> any eq 20

access-list 110 permit tcp host <navy-cash node #2 IP> any eq 21

access-list 110 permit tcp host <navy-cash node #2 IP> any eq 22

access-list 110 permit tcp host <navy-cash node #1 IP> any eq 80

access-list 110 permit tcp host <navy-cash node #2 IP> any eq 80

access-list 110 permit tcp host <navy-cash node #1 IP> any gt 1023

access-list 110 permit tcp host <navy-cash node #2 IP> any gt 1023

access-list 110 permit tcp host <navy-cash node #1 IP> any https 443

access-list 110 permit tcp host <navy-cash node #2 IP> any https 443

NOTE: When you apply these statements, be sure that they do not get placed below any deny statements. Also be sure that 

you use the correct greater than (gt) and equal to (eq) command for each specific line.
4 | Page

Failover Cluster Issues for all Navy Cash Versions

The following instructions are detailed in nature. If you’re hesitant to perform any of the following, please consult with your 

Shipboard IT or open a case with CSU.

If the ATMs and POS devices will not come online and the Disbursing Application and IEOD application can’t connect,

check the following to see if all the cluster items are online (use the below images as a reference 1471/3 and 1474):

 

1) Open the Failover Cluster Manager

xxxxxx 

xxxxxx

xxxxxx xxxxxx 

xxxxxx
5 | Page

2) Expand the left side as follows and select ‘Navy Cash’:

a. ncsvr-cl.ncsvr-d.navycash.navy.mil

I. Services and Applications

1. Navy Cash

3) Take note of the ‘Current Owner’ as that will determine where you need to check the Event Viewer and/or perform the

COAF Check.

a. If n1-xxxxxx is showing as the current owner, be sure you are logged in and perform the steps below on 

Node 1 (where xxxxxx is your shipUIC)

b. If n2-xxxxxx is showing as the current owner, be sure you are logged in and perform the steps below on 

Node 2 (where xxxxxx is your shipUIC)

c. To validate what server you are on perform thefollowing:

I. Open a command prompt

II. When the command prompt box appears, enterthe command: hostname

III. You should get a response of either n1-xxxxxx or n2-xxxxxx (where xxxxxx is your ship UIC)

4) Check to see if ncsvr-o (top item under Server Name) along with NCP and NavyCash Messaging Service are online 

(under Other Resources)

a. If ncsvr-o, NCP and NavyCash Messaging Service are offline proceed to the COAF Check section below

b. If ncsvr-o is online but NCP and NavyCash Messaging Service are offline, minimize the Failover Cluster Manager

and open the Event Viewer

5) In Event Viewer select Windows Logs on the left hand side and observer the log sizes displayed in the center panel 

for Application, Security and System (refer to image on nextpage)

6) If none of the log sizes are above 512MB then:

a. Attempt to transfer control the cluster from the current owner to the other node via the Failover Cluster Manager

I. If the transfer of the cluster was successful and all the services started on the other node, then perform a 

controlled restart (Start -> Restart) of the node that previously hadcontrol

II. If the transfer of the cluster failed, then proceed to the ‘Cluster Servers Restart Instructions’ above

7) If any of sizes are above 512MB then perform the following (this can only be performed on the Cluster Owner):

a. Minimize the Event Viewer and complete step 2 of COAF Clear instructions on the next page

b. Go back to the EventViewer, press F5 to refresh the current view. The file sizes should have dropped to almost 

nothing

c. Minimize the Event Viewer and go back to the Failover ClusterManager

d. Right click on NCP and select ‘Bring this resourceonline’

e. Wait for NCP to come online and then right click on NavyCash Messaging Service and select ‘Bring thisresource 

online’

I. If NCP and/or NavyCash Messaging Service will still not comeonline

1. Attempt to transfer control the cluster from the current owner to the othernode

a. If the transfer of the cluster was successful and all the services started on the other 

node, then perform a controlled restart (Start -> Restart) of the nodethat previously 

had control

b. If the transfer of the cluster failed, then proceed to the ‘Cluster Servers Restart 

Instructions’ above

II. If NCP and NavyCash Messaging Service have come online, perform a cluster move to ensure the other 

node works and all services comeonline

1. Repeat the instruction from the ‘Failover Cluster Issues’ section if needed for the othernode
6 | Page

COAF Check for all Navy Cash Versions

(Crash on Audit Failure)

If you saw the results of 4a from the Failover Cluster Issues section, then there is a chance that the server stopped certain services 

due to a ‘Crash on Audit Failure’. In order to check for and correct this issue, you will need to perform the following actions on the 

cluster’s Current Owner (as described in Failover Cluster Issues step3)

1) Open Event Viewer, select ‘Windows Logs’ on the left hand side and observer the log sizes displayed in the center panel for 

Application, Security and System

2) If any of sizes are above 512MB then perform thefollowing:

a. Check your server desktop for COAF_CLEAR.bat, if present, skip to step f

b. Go to S0.navycashupdate.navycash.fiscal.treasury.gov/rs/scripts webpage

c.

d. Download COAF_CLEAR.txt to the server desktop

e. On the desktop right click on “COAF_CLEAR.txt” and rename it to “COAF_CLEAR.bat”

f. Right click on it and run as admin

g. You will see a command prompt screen pop-up and run some commands, this will automatically archive all logs and 

reset the COAF flag. There is no harm in running this, however if you have to run often, NCTS should be notified to 

check your system

3) Go back to the Failover Cluster Manager

4) Right click on NCP and select ‘Bring this resource online’

5) Wait for NCP to come online and then right click on NavyCash Messaging Service and select ‘Bring this resource online’

a. If NCP and/or NavyCash Messaging Service will still not comeonline

i. Attempt to transfer control the cluster from the current owner to the othernode

1. If the transfer of the cluster was successful and all the services started on the other node, then 

perform a controlled restart (Start -> Restart) of the node that previously hadcontrol

2. If the transfer of the cluster failed, then preform the ‘Cluster Servers Restart Instructions’

b. If NCP and NavyCash Messaging Service have come online, perform a cluster move to ensure the other node works 

and all services come online

i. Repeat the instruction from the ‘Failover Cluster Issues’ section if needed for the otherNode

1. Note you might have to login to the virtual(s) if on 1474 and check/fix COAF there too
7 | Page

Round Trip Restart Aid for all Navy Cash Versions

If your ship has not had a round trip in over 24 hours, please perform the following troubleshooting steps after restart above:

The following needs to be done from the ACTIVE node. Whichever server you see the "F" drive on is the ACTIVE server.

1) Check and keep in mind what time the Navy Cash server says itis.

2) On the active server, go to the following location in 'My Computer'

A. F:\files\navydata\batproc

3) With this folder open, see what files (if any) are present. Generally speaking, one of the following scenarios will be what 

you see (ignore anything that doesn’t match the exact file names below, i.e. batch_process.sta.old doesn’t count):

A. You don't see any files.

B. You only see the file named 'batch_process.sta'

C. You see three files named 'batch_process.sta', 'file.pid' and 'sqlnet.output'

4) Open the 'Task Scheduler' program on the desktop and expand the left side out until you can select Navy Cash. Once 

selected, press the refresh button on the right hand side of thescreen.

5) Look for the task called 'Navy Cash Batch' to see what its status is. If the minutes on the server are 00, 20, or 40, then the 

task should be running. Depending on what step it is on, it could run for a few minutes before it ends.

6) Based on what you saw in steps 3 and 5 above will determine what action totake.

A. If in step 3 you saw the scenario for 3A or 3B and the Navy Cash Batch was running after refreshing it, then:

− Click on the 'Navy Cash Batch'

− On the right hand side, press'End'

− Note: Make sure there is no current RT batch file in F:\files\navydata\Work\ folder before ending the job.

− On the right hand side, press'Run'

− You should now see the files listed in 3C and the round trips should be good again and will need some 

time to catch up.

B. If in step 3 you saw the scenario for 3C AND the date/time for the file.pid and sqlnet.output is close to the servers 

time AND the Navy Cash Batch was running after refreshing it,then:

− Let it continue to run as it is processing something for the round trip.

C. If in step 3 you saw the scenario for 3C AND the date/time for the file.pid and sqlnet.output is over an hour old 

when compared to the servers time AND the Navy Cash Batch was running after refreshing it, then:

− Click on the 'Navy Cash Batch'

− On the right hand side, press'End'

− Note: Make sure there is no current RT batch file in F:\files\navydata\Work\ folder before ending the job.

− Switch back to the window where the F:\files\navydata\batproc directory is beingdisplayed

− Delete the files (if needed)

• file.pid

• sqlnet.output

− Switch back to the 'Task Scheduler'

− Click on the 'Navy Cash Batch'

− On the right hand side, press'Run'

− Both file.pid and sqlnet.output should re-appear. You should now see all the files listed from 3C and the 

round trips should be good again and will need some time to catchup.

7) Once the 'Navy Cash Batch' task is done, you should either see the scenario for 3A or 3B.

A. If you see 3A now, it means that the round trip has completed on the ship and is either between batches or 

there are no new batches to process.

B. If you see 3B now, it means that the round trip process has started but has not completed yet.
8 | Page

Password Rotation Aid for all Navy Cash Versions

(NEVER use Ctrl-Alt-Del through Windows)

Follow these steps to rotate server/node passwords

NOTE: You need Internet access for password rotation to work, if Ship has an internet outage that will need to be 

resolved first. See if you can reach NavyCash.com from Node 1 to verify.

1) Login to Node 1.

2) Check Cluster Manager that Node 1 is in control of the cluster.

3) Ensure Node 2 is up, but not in control of the cluster (note if you are only running 1 Node due to a

hardware/software issue, you will not be able to rotate your passwords).

4) Open Password Rotation Tool using icon on Node 1 desktop

5) Click on Get Current Status button.

6) If the Processing bar is red, this alerts you that something needs addressed before rotation can take place. (Some 

examples are SSL errors, or notice that Node 2 is not online)

 a. Open a case with CSU if an error occurs. Please include error message in your communication to CSU.

7) If everything connects properly, the processing status bar will be green which lets you know you can proceed to 

rotate PW. 

a.

b.

8) Click Rotate Password button. This process can take a minute or so. 

9) Write down and securely store new passwords.
9 | Page

Case Opening Guidance

When opening a case, please include as much information as possible:

-Any conditions leading up to Issue (power outage, network outage or upgrade etc.)

-What device or devices are affected (1 CAD, All POS, Workstation1, etc.)

Note: To find out what the device name for Windows devices, type cmd in start menu, once the 

Command prompt window opens, type: hostname

-If account issue, what account:

Server Account – nc-admin, ncinstall, ncship-admin

Workstation account to login to Windows

Disbursing application account

NavyCash.com website account