Repository: Bitmessage/PyBitmessage

Issue:
 list index out of range traceback when no connects found or file corrupt.
random choice need a list to Generates a random sample from a list, but in connection pool it find self.stream empty and through  list index out of range traceback:

Exception in thread Asyncore:
Traceback (most recent call last):
PyBitmessage/src/helper_random.py", line 65, in randomchoice
    return random.choice(population)  # nosec
  File "/usr/lib/python2.7/random.py", line 275, in choice
    return seq[int(self.random() * len(seq))]  # raises IndexError if seq is empty
IndexError: list index out of range

2018-08-09 13:28:22,901 - WARNING - No notification.sound plugin found


Discussion:
Looks like when `knownnodes` is empty, an exception is thrown.
I suspect that it caused by wrong import order in `bitmessagemain`. Can you please provide a steps to reproduce?
> knownnodes is empty

yes. 
this happens, when knownnodes.dat is empty or corrupted*,
but parsing dont raise an exception
to trigger createDefaultKnownNodes.

patch:
in knownnodes.py
after createDefaultKnownNodes()
insert:
```
    node_count = 0
    for stream in knownNodes.keys():
        for node in knownNodes[stream].keys():
            node_count += 1
    if node_count == 0:
        createDefaultKnownNodes()
    else:
        print("restored %i nodes" % node_count)

```

*corrupted?
blame the file, or blame the parser?
see knownnodes.py
```
def pickle_deserialize_old_knownnodes(source):
    knownNodes = pickle.load(source)
    for stream in knownNodes.keys():
        for node, params in knownNodes[stream].items():
            if isinstance(params, (float, int)):
                addKnownNode(stream, node, params)
```
the if condition is never satisfied,
so old nodes are not added.

somehow, old pickle and new json format got mixed up.

in my pickle-coded knownnodes.dat
params is a dict, and looks like this
```{'rating': 0, 'self': False, 'lastseen': 1538113792}```

i suggest:
1. check for duplicate nodes
2. add elif condition
```
def pickle_deserialize_old_knownnodes(source):
    knownNodes = pickle.load(source)
    for stream in knownNodes.keys():
        for node, params in knownNodes[stream].items():
            # find duplicate nodes
            found_node = False
            for old_stream in knownNodes.keys():
                for old_node in knownNodes[old_stream].keys():
                    if old_node == node:
                        found_node = True
                        break
                if found_node:
                    break
            if found_node:
                continue

            if isinstance(params, (float, int)):
                # add old format
                addKnownNode(stream, node, params)
            elif isinstance(params, dict):
                # add new format
                knownNodes[stream][peer] = params
```
ps, have a look at El Paquete Semanal,
for 'more' decentral file sharing.
our main problem is,
we still depend on 'their' infrastructure.
see also Freifunk.
would love to see 'bridges' like XMPP transports.
kinda like a high-latency broadcast protocol.

hope to help ^^

ps2, still waiting for bm to connect ....
it did fall back to the nine hard-coded DEFAULT_NODES,
but none shows up in network status.
ping-ing the nodes does work.


@milahu Too many questions and unrelated stuff (: 

Do you indeed update from BM < 6 so it has an old pickle knownnodes format? Could you please provide such file?

what a problem to import zero nodes from empty list? Yes default nodes should be added in this case. Though if the `knownnodes.dat` is really empty file it crashes with `EOFError`

No need to deduplicate dict because its keys are unique. What `old_stream`, did you implement streams?
> BM < 6

no .... i guess it was 0.6.3.2

in the logfile, i could not find the bm version.
in debug.py line 130 insert before 'if msg:'

        import version
        logger.log(logging.WARNING, "starting %s-%s" % (
            version.softwareName, version.softwareVersion))

my knownnodes.dat.bak has around 700kB with around 6k nodes.
sounds much to me ....

here is a sample file, with randomized ip addresses.
[knownnodes.dat.bak.2.gz](https://github.com/Bitmessage/PyBitmessage/files/2435630/knownnodes.dat.bak.2.gz)
created with
[my_pickle.py.txt](https://github.com/Bitmessage/PyBitmessage/files/2435640/my_pickle.py.txt)

but my last patch dont work.
all nodes seem duplicates.
that is, because pickle_deserialize_old_knownnodes
does overwrite knownNodes

knownnodes.py should look like this
```
def pickle_deserialize_old_knownnodes(source):
    new_knownNodes = pickle.load(source)
    for stream in new_knownNodes.keys():
        for node, params in new_knownNodes[stream].items():
            # find duplicate nodes
            found_node = False
            for old_stream in knownNodes.keys():
                for old_node in knownNodes[old_stream].keys():
                    if old_node == node:
                        found_node = True
                        print("unpickle: skip %s" % repr(node))
                        break
                if found_node:
                    break
            if found_node:
                continue

            if isinstance(params, (float, int)):
                # add old format
                addKnownNode(stream, node, params)
            elif isinstance(params, dict):
                # add new format
                knownNodes[stream][node] = params

```

why i still look for duplicates?
cos knownNodes has 2 keys, stream and node.
so one node can be in multiple streams.

old_stream and old_node are indices for the 'old' knownNodes,
where stream and node are indices for the 'new' new_knownNodes.

and ta-daa, its working : D
restored all 6k nodes + is connecting
im happy.
#1336 is an attempt to fix it but it's wrong. See my comments there. I haven't tried reproducing it myself, but what could happen is that all known nodes expire when your internet is down.

If the list is empty, it should re-bootsrap in my opinion.
Oh, you trying to use `knownnodes.dat` populated with random IPs. What you expect?
I see no point to repopulate dict with its values. Because for the latest pickle format `pickle.load()` should be sufficient.
> If the list is empty, it should re-bootsrap in my opinion.

no. if my `knownnodes.dat` has 6k nodes, but zero are loaded,
there must be something wrong with the parser.

> knownnodes.dat populated with random IPs

that was a sample file, to test parsing.
i have the original file here, did apply the patch above, and it works.
all nodes are loaded, bm connects, and stores nodes in new json format.

but i still have no idea, why it did store in pickle format.

> repopulate dict with its values

i only let you know, how i did the ip randomization.

as i said, im happy. do what you want.
There are actually two problems. The first one is in parsing. The second one is that if knownnodes for whatever reason becomes empty, the Asyncore loop will start consuming a lot of CPU.

Resolving patch:
diff --git a/src/class_singleCleaner.py b/src/class_singleCleaner.py
--- a/src/class_singleCleaner.py
+++ b/src/class_singleCleaner.py
@@ -55,7 +55,7 @@ def run(self):
             ) + (
                 float(BMConfigParser().get(
                     'bitmessagesettings', 'stopresendingafterxmonths')) *
-                (60 * 60 * 24 * 365)/12)
+                (60 * 60 * 24 * 365) / 12)
         except:
             # Either the user hasn't set stopresendingafterxdays and
             # stopresendingafterxmonths yet or the options are missing
@@ -96,9 +96,8 @@ def run(self):
                     "SELECT toaddress, ackdata, status FROM sent"
                     " WHERE ((status='awaitingpubkey' OR status='msgsent')"
                     " AND folder='sent' AND sleeptill<? AND senttime>?)",
-                    int(time.time()),
-                    int(time.time())
-                    - shared.maximumLengthOfTimeToBotherResendingMessages
+                    int(time.time()), int(time.time()) -
+                    shared.maximumLengthOfTimeToBotherResendingMessages
                 )
                 for row in queryreturn:
                     if len(row) < 2:
@@ -115,54 +114,28 @@ def run(self):
                     elif status == 'msgsent':
                         resendMsg(ackData)
 
-            # cleanup old nodes
-            now = int(time.time())
-
-            with knownnodes.knownNodesLock:
-                for stream in knownnodes.knownNodes:
-                    keys = knownnodes.knownNodes[stream].keys()
-                    for node in keys:
-                        try:
-                            # scrap old nodes
-                            if now - knownnodes.knownNodes[stream][node]["lastseen"] > 2419200: # 28 days
-                                shared.needToWriteKnownNodesToDisk = True
-                                del knownnodes.knownNodes[stream][node]
-                                continue
-                            # scrap old nodes with low rating
-                            if now - knownnodes.knownNodes[stream][node]["lastseen"] > 10800 and knownnodes.knownNodes[stream][node]["rating"] <= knownnodes.knownNodesForgetRating:
-                                shared.needToWriteKnownNodesToDisk = True
-                                del knownnodes.knownNodes[stream][node]
-                                continue
-                        except TypeError:
-                            print "Error in %s" % node
-                    keys = []
-
-            # Let us write out the knowNodes to disk
-            # if there is anything new to write out.
-            if shared.needToWriteKnownNodesToDisk:
-                try:
-                    knownnodes.saveKnownNodes()
-                except Exception as err:
-                    if "Errno 28" in str(err):
-                        logger.fatal(
-                            '(while receiveDataThread'
-                            ' knownnodes.needToWriteKnownNodesToDisk)'
-                            ' Alert: Your disk or data storage volume'
-                            ' is full. '
-                        )
-                        queues.UISignalQueue.put((
-                            'alert',
-                            (tr._translate("MainWindow", "Disk full"),
-                             tr._translate(
-                                 "MainWindow",
-                                 'Alert: Your disk or data storage volume'
-                                 ' is full. Bitmessage will now exit.'),
-                                True)
-                            ))
-                        # FIXME redundant?
-                        if shared.daemon or not state.enableGUI:
-                            os._exit(0)
-                shared.needToWriteKnownNodesToDisk = False
+            try:
+                # Cleanup knownnodes and handle possible severe exception
+                # while writing it to disk
+                knownnodes.cleanupKnownNodes()
+            except Exception as err:
+                if "Errno 28" in str(err):
+                    logger.fatal(
+                        '(while writing knownnodes to disk)'
+                        ' Alert: Your disk or data storage volume is full.'
+                    )
+                    queues.UISignalQueue.put((
+                        'alert',
+                        (tr._translate("MainWindow", "Disk full"),
+                         tr._translate(
+                             "MainWindow",
+                             'Alert: Your disk or data storage volume'
+                             ' is full. Bitmessage will now exit.'),
+                            True)
+                    ))
+                    # FIXME redundant?
+                    if shared.daemon or not state.enableGUI:
+                        os._exit(1)
 
 #            # clear download queues
 #            for thread in threading.enumerate():
@@ -206,8 +179,9 @@ def resendPubkeyRequest(address):
         pass
 
     queues.UISignalQueue.put((
-         'updateStatusBar',
-         'Doing work necessary to again attempt to request a public key...'))
+        'updateStatusBar',
+        'Doing work necessary to again attempt to request a public key...'
+    ))
     sqlExecute(
         '''UPDATE sent SET status='msgqueued' WHERE toaddress=?''',
         address)
diff --git a/src/knownnodes.py b/src/knownnodes.py
--- a/src/knownnodes.py
+++ b/src/knownnodes.py
@@ -1,13 +1,17 @@
+"""
+Manipulations with knownNodes dictionary.
+"""
+
 import json
 import os
 import pickle
-# import sys
 import threading
 import time
 
 import state
 from bmconfigparser import BMConfigParser
 from debug import logger
+from helper_bootstrap import dns
 
 knownNodesLock = threading.Lock()
 knownNodes = {stream: {} for stream in range(1, 4)}
@@ -105,10 +109,6 @@ def readKnownNodes():
         createDefaultKnownNodes()
 
     config = BMConfigParser()
-    # if config.safeGetInt('bitmessagesettings', 'settingsversion') > 10:
-    #     sys.exit(
-    #         'Bitmessage cannot read future versions of the keys file'
-    #         ' (keys.dat). Run the newer version of Bitmessage.')
 
     # your own onion address, if setup
     onionhostname = config.safeGet('bitmessagesettings', 'onionhostname')
@@ -157,3 +157,43 @@ def trimKnownNodes(recAddrStream=1):
         )[:knownNodesTrimAmount]
         for oldest in oldestList:
             del knownNodes[recAddrStream][oldest]
+
+
+def cleanupKnownNodes():
+    """
+    Cleanup knownnodes: remove old nodes and nodes with low rating
+    """
+    now = int(time.time())
+    needToWriteKnownNodesToDisk = False
+    dns_done = False
+
+    with knownNodesLock:
+        for stream in knownNodes:
+            keys = knownNodes[stream].keys()
+            if len(keys) <= 1 and not dns_done:  # leave at least one node
+                dns()
+                dns_done = True
+                continue
+            for node in keys:
+                try:
+                    # scrap old nodes
+                    if (now - knownNodes[stream][node]["lastseen"] >
+                            2419200):  # 28 days
+                        needToWriteKnownNodesToDisk = True
+                        del knownNodes[stream][node]
+                        continue
+                    # scrap old nodes with low rating
+                    if (now - knownNodes[stream][node]["lastseen"] > 10800 and
+                        knownNodes[stream][node]["rating"] <=
+                            knownNodesForgetRating):
+                        needToWriteKnownNodesToDisk = True
+                        del knownNodes[stream][node]
+                        continue
+                except TypeError:
+                    logger.warning('Error in %s', node)
+            keys = []
+
+    # Let us write out the knowNodes to disk
+    # if there is anything new to write out.
+    if needToWriteKnownNodesToDisk:
+        saveKnownNodes()


Test patch: