Repository: RocketMap/RocketMap

Issue:
speedscan workers moving to the same step
<!--
  
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!

  Join discord (channel: #help) for support at this link: https://discord.gg/PWp2bAm

-->

<!--- Provide a general summary of the issue in the Title above -->

## Expected Behavior

Speedscan workers should deconflict with each other before deciding to move to a step, otherwise they waste time moving there to just discover another worker has already beaten them to it.

<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->

## Current Behavior

Multiple speedscan workers move to same step. sometimes up to 90% of workers are moving to same step. here is example, but sometimes it's far more workers. 

This happens for TTH, spawn and bands (not just TTH as per example below).

```
Worker ID  | Start | xx| Proxy | Success | Failed | Empty | Skipped | Captchas | Message
Worker 010 | 15:06 | xx| No    |      49 |      8 |     0 |       0 |        0 | Moving 1172m to step 699 for a TTH.
...
Worker 014 | 15:06 | xx| No    |      67 |      6 |     0 |       0 |        0 | Moving 1077m to step 699 for a TTH.
...
Worker 028 | 14:41 | xx| No    |      89 |     15 |     1 |       0 |        0 | Moving 971m to step 699 for a TTH.
...
Worker 030 | 14:41 | xx| No    |      93 |     17 |     0 |       0 |        0 | Moving 947m to step 699 for a TTH.
```

<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->

## Possible Solution

Have each worker deconflict where they are moving before they start moving, and pick a different destination.

<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->

## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
1.  Run using -speed parameter with multiple workers. I am using -kph 35 -st 16 with 32 workers.
2.
3.
4.

## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used:
* Environment name and version (e.g. Python 2.7):
* Operating System and version (desktop or mobile):



Discussion:
Have same issue
I assume this can add some randomness to the workers' movement on map?

Resolving patch:
diff --git a/pogom/schedulers.py b/pogom/schedulers.py
--- a/pogom/schedulers.py
+++ b/pogom/schedulers.py
@@ -51,6 +51,8 @@
 import json
 import time
 import sys
+from timeit import default_timer
+from threading import Lock, current_thread
 from copy import deepcopy
 import traceback
 from collections import Counter
@@ -493,7 +495,9 @@ def __init__(self, queues, status, args):
         self.spawn_percent = []
         self.status_message = []
         self.tth_found = 0
+        # Initiate special types.
         self._stat_init()
+        self._locks_init()
 
     def _stat_init(self):
         self.spawns_found = 0
@@ -502,6 +506,9 @@ def _stat_init(self):
         self.scans_missed = 0
         self.scans_missed_list = []
 
+    def _locks_init(self):
+        self.lock_next_item = Lock()
+
     # On location change, empty the current queue and the locations list
     def location_changed(self, scan_location, db_update_queue):
         super(SpeedScan, self).location_changed(scan_location, db_update_queue)
@@ -859,129 +866,173 @@ def schedule(self):
 
     # Find the best item to scan next
     def next_item(self, status):
-        # score each item in the queue by # of due spawns or scan time bands
-        # can be filled
-
-        while not self.ready:
-            time.sleep(1)
-
-        now_date = datetime.utcnow()
-        now_time = time.time()
-        n = 0  # count valid scans reviewed
-        q = self.queues[0]
-        ms = (now_date - self.refresh_date).total_seconds() + self.refresh_ms
-        best = {'score': 0}
-        cant_reach = False
-        worker_loc = [status['latitude'], status['longitude']]
-        last_action = status['last_scan_date']
-
-        # check all scan locations possible in the queue
-        for i, item in enumerate(q):
-            # if already claimed by another worker or done, pass
-            if item.get('done', False):
-                continue
-
-            # if already timed out, mark it as Missed and check next
-            if ms > item['end']:
-                item['done'] = 'Missed' if not item.get(
-                    'done', False) else item['done']
-                continue
+        # Thread safety: don't let multiple threads get the same "best item".
+        with self.lock_next_item:
+            # Score each item in the queue by # of due spawns or scan time
+            # bands can be filled.
+
+            while not self.ready:
+                time.sleep(1)
+
+            now_date = datetime.utcnow()
+            now_time = time.time()
+            n = 0  # count valid scans reviewed
+            q = self.queues[0]
+            ms = ((now_date - self.refresh_date).total_seconds() +
+                  self.refresh_ms)
+            best = {'score': 0}
+            cant_reach = False
+            worker_loc = [status['latitude'], status['longitude']]
+            last_action = status['last_scan_date']
+
+            # Check all scan locations possible in the queue.
+            for i, item in enumerate(q):
+                # If already claimed by another worker or done, pass.
+                if item.get('done', False):
+                    continue
+
+                # If the item is parked by a different thread, or by a
+                # different account (should never happen, but making sure),
+                # pass.
+                our_parked_name = (current_thread().name + '-' +
+                                   status['username'])
+                if (item.get('thread_name', False) and
+                        item.get('thread_name') != our_parked_name):
+                    # We use 'parked_last_update' to determine when the
+                    # last time was since the thread passed the item with the
+                    # same thread name & username. If it's been too long, unset
+                    # the park so another worker can pick it up.
+                    now = default_timer()
+                    max_parking_idle_seconds = 3 * 60
+
+                    if (now - item.get('parked_last_update', now)
+                            > max_parking_idle_seconds):
+                        # Unpark & don't skip it.
+                        delattr(item, 'thread_name')
+                        delattr(item, 'parked_last_update')
+                    else:
+                        # Still parked. Skip it.
+                        continue
 
-            # if we just did a fresh band recently, wait a few seconds to space
-            # out the band scans
-            if now_date < self.next_band_date:
-                continue
+                # If already timed out, mark it as Missed and check next.
+                if ms > item['end']:
+                    item['done'] = 'Missed' if not item.get(
+                        'done', False) else item['done']
+                    continue
+
+                # If we just did a fresh band recently, wait a few seconds to
+                # space out the band scans.
+                if now_date < self.next_band_date:
+                    continue
+
+                # If the start time isn't yet, don't bother looking further,
+                # since queue sorted by start time.
+                if ms < item['start']:
+                    break
+
+                loc = item['loc']
+                distance = equi_rect_distance(loc, worker_loc)
+                secs_to_arrival = distance / self.args.kph * 3600
+
+                # If we can't make it there before it disappears, don't bother
+                # trying.
+                if ms + secs_to_arrival > item['end']:
+                    cant_reach = True
+                    continue
+
+                n += 1
+
+                # Bands are top priority to find new spawns first
+                score = 1e12 if item['kind'] == 'band' else (
+                    1e6 if item['kind'] == 'TTH' else 1)
+
+                # For spawns, score is purely based on how close they are to
+                # last worker position
+                score = score / (distance + .01)
+
+                if score > best['score']:
+                    best = {'score': score, 'i': i}
+                    best.update(item)
+
+            prefix = 'Calc %.2f for %d scans:' % (time.time() - now_time, n)
+            loc = best.get('loc', [])
+            step = best.get('step', 0)
+            i = best.get('i', 0)
+            messages = {
+                'wait': 'Nothing to scan.',
+                'early': 'Early for step {}; waiting a few seconds...'.format(
+                    step),
+                'late': ('API response on step {} delayed by {} seconds. ' +
+                         'Possible causes: slow proxies, internet, or ' +
+                         'Niantic servers.').format(
+                             step,
+                             int((now_date - last_action).total_seconds())),
+                'search': 'Searching at step {}.'.format(step),
+                'invalid': ('Invalid response at step {}, abandoning ' +
+                            'location.').format(step)
+            }
 
-            # if the start time isn't yet, don't bother looking further, since
-            # queue sorted by start time
-            if ms < item['start']:
-                break
+            try:
+                item = q[i]
+            except IndexError:
+                messages['wait'] = ('Search aborting.'
+                                    + ' Overseer refreshing queue.')
+                return -1, 0, 0, 0, messages
+
+            if best['score'] == 0:
+                if cant_reach:
+                    messages['wait'] = ('Not able to reach any scan'
+                                        + ' under the speed limit.')
+                return -1, 0, 0, 0, messages
 
-            loc = item['loc']
             distance = equi_rect_distance(loc, worker_loc)
-            secs_to_arrival = distance / self.args.kph * 3600
-
-            # if we can't make it there before it disappears, don't bother
-            # trying
-            if ms + secs_to_arrival > item['end']:
-                cant_reach = True
-                continue
-
-            n += 1
-
-            # Bands are top priority to find new spawns first
-            score = 1e12 if item['kind'] == 'band' else (
-                1e6 if item['kind'] == 'TTH' else 1)
-
-            # For spawns, score is purely based on how close they are to last
-            # worker position
-            score = score / (distance + .01)
-
-            if score > best['score']:
-                best = {'score': score, 'i': i}
-                best.update(item)
-
-        prefix = 'Calc %.2f for %d scans:' % (time.time() - now_time, n)
-        loc = best.get('loc', [])
-        step = best.get('step', 0)
-        i = best.get('i', 0)
-        messages = {
-            'wait': 'Nothing to scan.',
-            'early': 'Early for step {}; waiting a few seconds...'.format(
-                step),
-            'late': ('API response on step {} delayed by {} seconds. ' +
-                     'Possible causes: slow proxies, internet, or ' +
-                     'Niantic servers.').format(
-                         step, int((now_date - last_action).total_seconds())),
-            'search': 'Searching at step {}.'.format(step),
-            'invalid': ('Invalid response at step {}, abandoning ' +
-                        'location.').format(step)
-        }
-
-        try:
-            item = q[i]
-        except IndexError:
-            messages['wait'] = 'Search aborting. Overseer refreshing queue.'
-            return -1, 0, 0, 0, messages
-
-        if best['score'] == 0:
-            if cant_reach:
-                messages['wait'] = ('Not able to reach any scan under the ' +
-                                    'speed limit.')
-            return -1, 0, 0, 0, messages
-
-        if (equi_rect_distance(loc, worker_loc) >
-                (now_date - last_action).total_seconds() *
-                self.args.kph / 3600):
-
-            messages['wait'] = 'Moving {}m to step {} for a {}.'.format(
-                int(equi_rect_distance(loc, worker_loc) * 1000), step,
-                best['kind'])
-            return -1, 0, 0, 0, messages
-
-        prefix += ' Step %d,' % (step)
-        # Check again if another worker heading there
-        if item.get('done', False):
-            messages['wait'] = ('Skipping step {}. Other worker already ' +
-                                'scanned.').format(step)
-            return -1, 0, 0, 0, messages
-
-        if not self.ready:
-            messages['wait'] = 'Search aborting. Overseer refreshing queue.'
-            return -1, 0, 0, 0, messages
-
-        # if a new band, set the date to wait until for the next band
-        if best['kind'] == 'band' and best['end'] - best['start'] > 5 * 60:
-            self.next_band_date = datetime.utcnow() + timedelta(
-                seconds=self.band_spacing)
-
-        # Mark scanned
-        item['done'] = 'Scanned'
-        status['index_of_queue_item'] = i
-
-        messages['search'] = 'Scanning step {} for a {}.'.format(
-            best['step'], best['kind'])
-        return best['step'], best['loc'], 0, 0, messages
+            if (distance >
+                    (now_date - last_action).total_seconds() *
+                    self.args.kph / 3600):
+                # Flag item as "parked" by a specific thread, because
+                # we're waiting for it. This will avoid all threads "walking"
+                # to the same item.
+                our_parked_name = (current_thread().name + '-' +
+                                   status['username'])
+                item['thread_name'] = our_parked_name
+
+                # CTRL+F 'parked_last_update' in this file for more info.
+                item['parked_last_update'] = default_timer()
+
+                messages['wait'] = 'Moving {}m to step {} for a {}.'.format(
+                    int(distance * 1000), step,
+                    best['kind'])
+                return -1, 0, 0, 0, messages
+
+            prefix += ' Step %d,' % (step)
+
+            # Check again if another worker heading there.
+            # TODO: Check if this is still necessary. I believe this was
+            # originally a failed attempt at thread safety, which still
+            # resulted in a race condition (multiple workers heading to the
+            # same spot). A thread Lock has since been added.
+            if item.get('done', False):
+                messages['wait'] = ('Skipping step {}. Other worker already ' +
+                                    'scanned.').format(step)
+                return -1, 0, 0, 0, messages
+
+            if not self.ready:
+                messages['wait'] = ('Search aborting.'
+                                    + ' Overseer refreshing queue.')
+                return -1, 0, 0, 0, messages
+
+            # If a new band, set the date to wait until for the next band.
+            if best['kind'] == 'band' and best['end'] - best['start'] > 5 * 60:
+                self.next_band_date = datetime.utcnow() + timedelta(
+                    seconds=self.band_spacing)
+
+            # Mark scanned
+            item['done'] = 'Scanned'
+            status['index_of_queue_item'] = i
+
+            messages['search'] = 'Scanning step {} for a {}.'.format(
+                best['step'], best['kind'])
+            return best['step'], best['loc'], 0, 0, messages
 
     def task_done(self, status, parsed=False):
         if parsed:


Test patch: