Repository: DataBiosphere/toil

Issue:
support shared temp directory in distributed system
Related to #1397

workers will attempt to kill pids belonging to jobs other workers if the same directory onthe same file system is used as TMPDIR across multiple nodes. This can be circumvented by adding a `machine-id` check to the process.


Discussion:
The fix for this is to add a method to `toil.common` that can generate a UUID specific to a machine. then in `toil.fileStore.AbstractFileSTore.__init__`  you add `self.machineID = getUniqueMachineID()`. This value needs to be added to the job state in both fileStores and then ensure `findAndHandleDeadJobs` only looks at jobs that have `jobState.machineID == self.machineID`.

A simple google search returns uuid.getnode() but I'm not convinced of its reliability. If someone can research this and convince me that it is reliable, I can implement this feature or guide them in the process.
I think the jobState (and all other local files) needs to be written to a machine unique location so that workers are not overwriting it. 
Logically, the jobState is a description of the job and it makes sense to keep it within the job's local directory. This also keeps the interface near-similar to that of the cachingFileStore. And honestly, implementing a machine-unique location is tantamount to generating a machine-unique ID that is checked so I would go with the one i suggested.
The problem occurs because there is no job local directory. All worker are writing into the same directory - the jobState file is being overwritten.
not really. There is a workflow directory that is being shared by all workers. Each worker creates a directory for its current job and
`CachingFileStore`: puts the jobstate info into the `_cacheState` file. This file can only be read by one process at a time because of the locking mechanism.
`NonCachingFileStore`: puts the jobstate there. This file is only written by the current job.

Currently each job on startup will look at all pids in the _cacheState file or the pids in the individual jobState files and then attempt to cleanup jobs that are dead (pid doesn't exist). When this feature is implemented, jobs will only cleanup behind other jobs spawned by the same node.

Fair enough. The worker walks the entire workDir looking for jobState files even those that don't belong to it. So yes, I think your fix should work. It will fix a big problem for me.
WE only need to figure out a reliable unique ID generator
MAC address?
Refer my above comment about `uuid.getnode()`. 

>Get the hardware address as a 48-bit positive integer. The first time this runs, it may launch a separate program, which could be quite slow. If all attempts to obtain the hardware address fail, we choose a random 48-bit number with its eighth bit set to 1 as recommended in RFC 4122. “Hardware address” means the MAC address of a network interface, and on a machine with multiple network interfaces the MAC address of any one of them may be returned.

This could be problematic. Also, we want to do this with the minimal use of external libraries and i think a system call (the other option according to stack overflow) is overkill.
@arkal [uuid.uuid4](https://docs.python.org/2/library/uuid.html#uuid.uuid4) generates a random uuid. This is the solution I will go for unless you suggest something different.

edit: nevermind I see that the uuid needs to be reproducible for the node. It serves the purpose of node id.
 Proposed solution:

Each node will have a uuid stored in `~/.toil/node_id`.

Each time `worker.py` runs it will check if this file exists. If it does, then the node will use it as the uuid. If it does not exist then it will generate a uuid and store it in the file in a race-safe way.

The rest is described above by @arkal.

Why this is better than MAC address:
MAC address is specific to network card which can be virtual and can change. 
I’m not clear on this. Are storing the UUID for each node in this file, i.e. multiple entries or is the UUID part of the filename or are you expecting ~/ to be unique on each node?

Evan.
--
Evan Thomas
Research Computing Scientist
Computational Biology
The Walter and Eliza Hall Institute of Medical Research
Mobile: +61423000246 | Skype: evanathomas | Email: thomas.e@wehi.edu.au<mailto:thomas.e@wehi.edu.au>





On 10 Feb 2017, at 6:36 am, jessebrennan <notifications@github.com<mailto:notifications@github.com>> wrote:


Proposed solution:

Each node will have a uuid stored in ~/.toil_node_id.

Each time worker.py runs it will check if this file exists. If it does, then the node will use it as the uuid. If it does not exist then it will generate a uuid and store it in the file in a race-safe way.

The rest is described above by @arkal<https://github.com/arkal>.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<https://github.com/BD2KGenomics/toil/issues/1462#issuecomment-278748590>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVADDYaZFrQko6UU0p77FlhYrHh3qFn-ks5ra2q_gaJpZM4LyBV5>.


Each node will look for a file called `~/.toil/nodeID`. If the file exists, it will read the uuid stored within that file and use that to describe itself. If it doesn't exist, it will create the file and store a uuid within it. This uuid will be used by every other worker that spawns on that node. The file should only be created ONCE on a machine if the user never deletes it. Toil will NEVER delete that file. Now unless all nodes on a cluster share the same home directory (Not the same path, but physically the same directory on the same disk), this should work.
Following up on @arkal's explanation, that uuid will be included in the .jobstate file along side the pid. `_pidExists` will check the file uuid against the uuid in .toil_node_id to insure the pid in the file refers to the pid on the correct node.

While we're at it, we might want to include the pid start time to insure the pid hasn't been reused, as @evan-wehi brought up in #1397. 
All our nodes share the same file system, including the head node.
--
Evan Thomas
Research Computing Scientist
Computational Biology
The Walter and Eliza Hall Institute of Medical Research
Mobile: +61423000246 | Skype: evanathomas | Email: thomas.e@wehi.edu.au<mailto:thomas.e@wehi.edu.au>





On 10 Feb 2017, at 9:59 am, Arjun Arkal Rao <notifications@github.com<mailto:notifications@github.com>> wrote:


Each node will look for a file called ~/.toil/nodeID. If the file exists, it will read the uuid stored within that file and use that to describe itself. If it doesn't exist, it will create the file and store a uuid within it. This uuid will be used by every other worker that spawns on that node. The file should only be created ONCE on a machine if the user never deletes it. Toil will NEVER delete that file. Now unless all nodes on a cluster share the same home directory (Not the same path, but physically the same directory on the same disk), this should work.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<https://github.com/BD2KGenomics/toil/issues/1462#issuecomment-278802855>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVADDYuKWFX_8hbLMlMpkz7CFPxblaLGks5ra5pdgaJpZM4LyBV5>.


@evan-wehi their entire file system is shared (except the small local tmpdir)?
Correct. Each node is a VM spun for the purpose and has a 50GB filesystem for the OS. Everything else is the same shared filesystems.


--
Evan Thomas
Research Computing Scientist
Computational Biology
The Walter and Eliza Hall Institute of Medical Research
Mobile: +61423000246 | Skype: evanathomas | Email: thomas.e@wehi.edu.au<mailto:thomas.e@wehi.edu.au>





On 10 Feb 2017, at 10:14 am, Christopher Ketchum <notifications@github.com<mailto:notifications@github.com>> wrote:


@evan-wehi<https://github.com/evan-wehi> their entire file system is shared (except the small local tmpdir)?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/BD2KGenomics/toil/issues/1462#issuecomment-278805959>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVADDTanetbXO-TINtcdTpmKHR6_M5o0ks5ra527gaJpZM4LyBV5>.


@evan-wehi does ~/ map to the same place on all machines?
Yes - all filesystems are shared - every node sees the same thing at the same location, so ~ is also the same for all nodes.


--
Evan Thomas
Research Computing Scientist
Computational Biology
The Walter and Eliza Hall Institute of Medical Research
Mobile: +61423000246 | Skype: evanathomas | Email: thomas.e@wehi.edu.au<mailto:thomas.e@wehi.edu.au>





On 10 Feb 2017, at 10:38 am, Arjun Arkal Rao <notifications@github.com<mailto:notifications@github.com>> wrote:


@evan-wehi<https://github.com/evan-wehi> does ~/ map to the same place on all machines?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/BD2KGenomics/toil/issues/1462#issuecomment-278811498>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVADDZt0HQis8JVC9sAm8dvPR7OSMKmBks5ra6NjgaJpZM4LyBV5>.


Can you use the ssh host key?


--
Evan Thomas
Research Computing Scientist
Computational Biology
The Walter and Eliza Hall Institute of Medical Research
Mobile: +61423000246 | Skype: evanathomas | Email: thomas.e@wehi.edu.au<mailto:thomas.e@wehi.edu.au>





On 10 Feb 2017, at 10:44 am, Evan Thomas <thomas.e@wehi.edu.au<mailto:thomas.e@wehi.edu.au>> wrote:

Yes - all filesystems are shared - every node sees the same thing at the same location, so ~ is also the same for all nodes.


--
Evan Thomas
Research Computing Scientist
Computational Biology
The Walter and Eliza Hall Institute of Medical Research
Mobile: +61423000246 | Skype: evanathomas | Email: thomas.e@wehi.edu.au<mailto:thomas.e@wehi.edu.au>





On 10 Feb 2017, at 10:38 am, Arjun Arkal Rao <notifications@github.com<mailto:notifications@github.com>> wrote:


@evan-wehi<https://github.com/evan-wehi> does ~/ map to the same place on all machines?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/BD2KGenomics/toil/issues/1462#issuecomment-278811498>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVADDZt0HQis8JVC9sAm8dvPR7OSMKmBks5ra6NjgaJpZM4LyBV5>.



forgive me if this is naive... where are you storing per-node ssh host keys if not in ~/.ssh ?
The host keys identifies the host, it is not related to the user’s keys. They are stored somewhere in the /etc (depending on your ssh implementation). You can get host the fingerprint, which derives from the key, with standard APIs. But it turns out all our host keys are the same :(. (We could probably change that though).

Why is hostname no good?

Evan.
--
Evan Thomas
Research Computing Scientist
Computational Biology
The Walter and Eliza Hall Institute of Medical Research
Mobile: +61423000246 | Skype: evanathomas | Email: thomas.e@wehi.edu.au<mailto:thomas.e@wehi.edu.au>





On 10 Feb 2017, at 11:26 am, Arjun Arkal Rao <notifications@github.com<mailto:notifications@github.com>> wrote:


forgive me if this is naive... where are you storing per-node ssh host keys if not in ~/.ssh ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/BD2KGenomics/toil/issues/1462#issuecomment-278821006>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVADDUo7M_fTq0Cv27xGHE5ZFozlqG2Vks5ra66jgaJpZM4LyBV5>.


hostname is an option if you have a static ip and/or you have clearly defined your hostname. This might make sense for large clusters but is not universally true. Most people don't setup a hostname on their laptops and the hostname can change with a change in the public IP.

Here's my suggestion.
Before we check a pid on a worker, we need to ensure the job belonged to that worker. This is done using a uuid that the worker receives in one of 3 ways in decreasing order of preference:
1. The user specifies a file path (that must exist on all nodes at the same place) that contains this value (think of this as a toilrc).
2. If the user does not specify a path, Toil will look for TOIL_NODE_ID in the environment variables for the node
3. If 1 and 2 don't pan out, toil will use python's uuid.getnode() that uses the mac address. This has kinks but it is the fallback. We expect this to give you the same value every time but if it doesn't, it just means that failed jobs won't be cleaned up. We will document this issue clearly so users know about it.
You may need to offer a variety of strategies which the user can choose from. For example hostname will work for usAs I understand it, this is required to clean up failed jobs. Is it needed for anything else? If the user cannot provide a mechanism then they may need to do the cleanup themselves. It seems that the cleanup is not guaranteed to run anyway, for example if a node fails and work is never rescheduled on it.



--
Evan Thomas
Research Computing Scientist
Computational Biology
The Walter and Eliza Hall Institute of Medical Research
Mobile: +61423000246 | Skype: evanathomas | Email: thomas.e@wehi.edu.au<mailto:thomas.e@wehi.edu.au>





On 10 Feb 2017, at 11:53 am, Arjun Arkal Rao <notifications@github.com<mailto:notifications@github.com>> wrote:


hostname is an option if you have a static ip and/or you have clearly defined your hostname. This might make sense for large clusters but is not universally true. Most people don't setup a hostname on their laptops and the hostname can change with a change in the public IP.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/BD2KGenomics/toil/issues/1462#issuecomment-278825563>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVADDWtxzBu8fLhwDLdY9eQgr_DMx0Djks5ra7UngaJpZM4LyBV5>.


Option 2.5 mutually exclusive group [--hostNameFile |--hostNameIDCommand] and you can provide a command that will be run with subprocess.get_output.
1. Presumably same path different filesystem?
2. How is this being set? /etc/profile?
3. Although I suggested earlier using the MAC address we may end up have at least one interface with the same address in each node (to get around the opposite problem. i.e. make all nodes look the same to deal with refractory licensed software).

I like option 2.5!!

Evan.
--
Evan Thomas
Research Computing Scientist
Computational Biology
The Walter and Eliza Hall Institute of Medical Research
Mobile: +61423000246 | Skype: evanathomas | Email: thomas.e@wehi.edu.au<mailto:thomas.e@wehi.edu.au>





On 10 Feb 2017, at 11:58 am, Arjun Arkal Rao <notifications@github.com<mailto:notifications@github.com>> wrote:


Here's my suggestion.
Before we check a pid on a worker, we need to ensure the job belonged to that worker. This is done using a uuid that the worker receives in one of 3 ways in decreasing order of preference:

  1.  The user specifies a file path (that must exist on all nodes at the same place) that contains this value (think of this as a toilrc).
  2.  If the user does not specify a path, Toil will look for TOIL_NODE_ID in the environment variables for the node
  3.  If 1 and 2 don't pan out, toil will use python's uuid.getnode() that uses the mac address. This has kinks but it is the fallback. We expect this to give you the same value every time but if it doesn't, it just means that failed jobs won't be cleaned up. We will document this issue clearly so users know about it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/BD2KGenomics/toil/issues/1462#issuecomment-278826319>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVADDXFAv3RFyiqxa7CYB2opKMP-AmIOks5ra7Y1gaJpZM4LyBV5>.


Ok We're looking into it! Thanks for the input!

Resolving patch:
diff --git a/src/toil/batchSystems/torque.py b/src/toil/batchSystems/torque.py
--- a/src/toil/batchSystems/torque.py
+++ b/src/toil/batchSystems/torque.py
@@ -20,6 +20,7 @@
 import time
 import math
 import sys
+import shlex
 import xml.etree.ElementTree as ET
 import tempfile
 
@@ -90,6 +91,13 @@ def getJobExitCode(self, torqueJobID):
                     status = line.split(' = ')[1]
                     logger.debug('Exit Status: ' + status)
                     return int(status)
+                if 'unknown job id' in line.lower():
+                    # some clusters configure Torque to forget everything about just
+                    # finished jobs instantly, apparently for performance reasons
+                    logger.debug('Batch system no longer remembers about job {}'.format(torqueJobID))
+                    # return assumed success; status files should reveal failure
+                    return 0
+            return None
 
         """
         Implementation-specific helper methods
@@ -106,7 +114,7 @@ def prepareQsub(self, cpu, mem, jobID):
             # qsubline = ['qsub', '-V', '-j', 'oe', '-o', '/dev/null',
             #             '-e', '/dev/null', '-N', 'toil_job_{}'.format(jobID)]
 
-            qsubline = ['qsub', '-V', '-N', 'toil_job_{}'.format(jobID)]
+            qsubline = ['qsub', '-S', '/bin/sh', '-V', '-N', 'toil_job_{}'.format(jobID)]
 
             if self.boss.environment:
                 qsubline.append('-v')
@@ -116,19 +124,35 @@ def prepareQsub(self, cpu, mem, jobID):
             reqline = list()
             if mem is not None:
                 memStr = str(mem / 1024) + 'K'
-                reqline += ['-l mem=' + memStr]
+                reqline.append('mem=' + memStr)
 
             if cpu is not None and math.ceil(cpu) > 1:
-                qsubline.extend(['-l ncpus=' + str(int(math.ceil(cpu)))])
-
-            # "Native extensions" for TORQUE (see DRMAA or SAGA)
-            nativeConfig = os.getenv('TOIL_TORQUE_ARGS')
-            if nativeConfig is not None:
-                logger.debug("Native TORQUE options appended to qsub from TOIL_TORQUE_RESOURCES env. variable: {}".format(nativeConfig))
-                if ("mem=" in nativeConfig) or ("nodes=" in nativeConfig) or ("ppn=" in nativeConfig):
-                    raise ValueError("Incompatible resource arguments ('mem=', 'nodes=', 'ppn='): {}".format(nativeConfig))
-            qsubline.extend(nativeConfig.split())
+                reqline.append('ncpus=' + str(int(math.ceil(cpu))))
+
+            # Other resource requirements can be passed through the environment (see man qsub)
+            reqlineEnv = os.getenv('TOIL_TORQUE_REQS')
+            if reqlineEnv is not None:
+                logger.debug("Additional Torque resource requirements appended to qsub from "\
+                        "TOIL_TORQUE_REQS env. variable: {}".format(reqlineEnv))
+                if ("mem=" in reqlineEnv) or ("nodes=" in reqlineEnv) or ("ppn=" in reqlineEnv):
+                    raise ValueError("Incompatible resource arguments ('mem=', 'nodes=', 'ppn='): {}".format(reqlineEnv))
+
+                reqline.append(reqlineEnv)
             
+            if reqline:
+                qsubline += ['-l',','.join(reqline)]
+            
+            # All other qsub parameters can be passed through the environment (see man qsub).
+            # No attempt is made to parse them out here and check that they do not conflict
+            # with those that we already constructed above
+            arglineEnv = os.getenv('TOIL_TORQUE_ARGS')
+            if arglineEnv is not None:
+                logger.debug("Native Torque options appended to qsub from TOIL_TORQUE_ARGS env. variable: {}".\
+                        format(arglineEnv))
+                if ("mem=" in arglineEnv) or ("nodes=" in arglineEnv) or ("ppn=" in arglineEnv):
+                    raise ValueError("Incompatible resource arguments ('mem=', 'nodes=', 'ppn='): {}".format(arglineEnv))
+                qsubline += shlex.split(arglineEnv)
+
             return qsubline
 
         def generateTorqueWrapper(self, command):
@@ -138,7 +162,7 @@ def generateTorqueWrapper(self, command):
             """
             _, tmpFile = tempfile.mkstemp(suffix='.sh', prefix='torque_wrapper')
             fh = open(tmpFile , 'w')
-            fh.write("$!/bin/bash\n\n")
+            fh.write("$!/bin/sh\n\n")
             fh.write("cd $PBS_O_WORKDIR\n\n")
             fh.write(command + "\n")
             fh.close
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -21,6 +21,7 @@
 import tempfile
 import time
 import socket
+import uuid
 from argparse import ArgumentParser
 from threading import Thread
 
@@ -567,6 +568,60 @@ def addGroup(headingString, bodyString):
         raise RuntimeError("Unanticipated class passed to addOptions(), %s. Expecting "
                            "argparse.ArgumentParser" % parser.__class__)
 
+def getNodeID(extraIDFiles=None):
+    """
+    Return unique ID of the current node (host).
+    Tries several methods until success. The returned ID should be identical across calls from different processes on 
+    the same node at least until the next OS reboot.
+
+    The last resort method is uuid.getnode() that in some rare OS configurations may return a random ID each time it is
+    called. However, this method should never be reached on a Linux system, because reading from 
+    /proc/sys/kernel/random/boot_id will be tried prior to that. If uuid.getnode() is reached, it will be called twice,
+    and exception raised if the values are not identical.
+
+    :param list extraIDFiles: Optional list of additional file names to try reading node ID before trying default 
+    methods. ID should be a single word (no spaces) on the first line of the file.
+    """
+    if extraIDFiles is None:
+        extraIDFiles = []
+    idSourceFiles = extraIDFiles + ["/var/lib/dbus/machine-id", "/proc/sys/kernel/random/boot_id"]
+    for idSourceFile in idSourceFiles:
+        if os.path.exists(idSourceFile):
+            try:
+                with open(idSourceFile, "r") as inp:
+                    nodeID = inp.readline().strip()
+            except EnvironmentError:
+                logger.warning(("Exception when trying to read ID file {}. Will try next method to get node ID").\
+                        format(idSourceFile), exc_info=True)
+            else:
+                if len(nodeID.split()) == 1:
+                    logger.debug("Obtained node ID {} from file {}".format(nodeID,idSourceFile))
+                    break
+                else:
+                    logger.warning(("Node ID {} from file {} contains spaces. Will try next method to get node ID").\
+                            format(nodeID, idSourceFile))
+    else:
+        nodeIDs = []
+        for i_call in range(2):
+            nodeID = str(uuid.getnode()).strip()
+            if len(nodeID.split()) == 1:
+                nodeIDs.append(nodeID)
+            else:
+                logger.warning("Node ID {} from uuid.getnode() contains spaces".format(nodeID))
+        nodeID = ""
+        if len(nodeIDs) == 2:
+            if nodeIDs[0] == nodeIDs[1]:
+                nodeID = nodeIDs[0]
+            else:
+                logger.warning("Different node IDs {} received from repeated calls to uuid.getnode(). You should use "\
+                        "another method to generate node ID.".format(nodeIDs))
+
+            logger.debug("Obtained node ID {} from uuid.getnode()".format(nodeID))
+    if not nodeID:
+        logger.warning("Failed to generate stable node ID, returning empty string. If you see this message with a "\
+                "work dir on a shared file system when using workers running on multiple nodes, you might experience "\
+                "cryptic job failures")
+    return nodeID
 
 class Toil(object):
     """
@@ -934,8 +989,9 @@ def getWorkflowDir(workflowID, configWorkDir=None):
         if not os.path.exists(workDir):
             raise RuntimeError("The directory specified by --workDir or TOIL_WORKDIR (%s) does not "
                                "exist." % workDir)
-        # Create the workflow dir
-        workflowDir = os.path.join(workDir, 'toil-%s' % workflowID)
+        # Create the workflow dir, make it unique to each host in case workDir is on a shared FS.
+        # This prevents workers on different nodes from erasing each other's directories.
+        workflowDir = os.path.join(workDir, 'toil-%s-%s' % (workflowID, getNodeID()))
         try:
             # Directory creation is atomic
             os.mkdir(workflowDir)
@@ -982,7 +1038,6 @@ def _assertContextManagerUsed(self):
         if not self._inContextManager:
             raise ToilContextManagerException()
 
-
 class ToilRestartException(Exception):
     def __init__(self, message):
         super(ToilRestartException, self).__init__(message)


Test patch:
diff --git a/src/toil/test/src/nodeIDTest.py b/src/toil/test/src/nodeIDTest.py
new file mode 100644
--- /dev/null
+++ b/src/toil/test/src/nodeIDTest.py
@@ -0,0 +1,34 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+from toil.common import getNodeID
+from toil.test import ToilTest
+
+class GetNodeIDTest(ToilTest):
+    def testIDStability(self):
+        prevNodeID = None
+        for i in range(10):
+            nodeID = getNodeID()
+            if i > 0:
+                self.assertEquals(nodeID,prevNodeID)
+            prevNodeID = nodeID
+
+    def testSkipNonExistingFile(self):
+        import uuid, os
+        no_file = "/etc/totally_non_existing_"+str(uuid.uuid4())
+        self.assertFalse(os.path.exists(no_file), "Could not make up a missing file name")
+        nodeID1 = getNodeID([no_file])
+        nodeID2 = getNodeID()
+        self.assertEquals(nodeID1,nodeID2)
+