Repository: RocketMap/RocketMap

Issue:
Webhooks getting delayed anywhere from 2 - 45+ minutes
<!--

  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!
  ISSUES ARE NOT FOR SUPPORT!!!

  Join discord (channel: #help) for support at this link: https://discord.gg/rocketmap
  
  Submit feature requests @ http://vote.devkat.org
  
-->

<!--- Provide a general summary of the issue in the Title above -->

## Expected Behavior
Should be getting sent based off of -whfi

## Current Behavior
webhooks sending extremely delayed, if at all. sending the post to webhook outside of RM works immediately. multiple reports on discord of same issue.

issue seemed to have started at some point around July 15th.

## Webhook settings used:
webhook: [http://127.0.0.1:5112]
wh-types:[pokemon] 
webhook-blacklist: [1,2,4,5,7,8,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,27,29,30,32,33,35,37,39,41,43,44,46,48,50,52,54,56,58,60,61,63,66,69,72,74,77,79,86,90,92,96,98,102,104,111,116,118,119,120,129,133,152,153,155,156,158,161,162,163,164,165,166,167,168,170,177,178,183,184,185,187,188,190,191,193,194,195,198,200,202,203,204,205,206,207,209,210,211,215,216,218,220,223,224,226,227,228,231,234]

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used: Latest
* Environment name and version (e.g. Python 2.7): python 2.7
* Operating System and version (desktop or mobile): Windows 7

![git status](https://user-images.githubusercontent.com/30736628/29011305-dfb03680-7aff-11e7-9e0d-cf5708f95ff2.png)
![negative time - alarm](https://user-images.githubusercontent.com/30736628/29011306-dfb33948-7aff-11e7-94de-70ff5596cf04.png)



Discussion:
@macedoniangrotty Could you include some logs and a screenshot of `git status && git log -1`?
updated
I can confirm this, I also notice that some webhooks are delayed :)
Same problem

Also confirming this problem. I thought at first that it was PokeAlarm's issue, but it seems like it's RM's fault.
Can everyone affected post their webhook settings? Make sure to filter out all identifying details (e.g. domain names).

I've downloaded @macedoniangrotty's log file and removed it from the original post in case it contains personal details. I'll look through it after I've received your settings.

I'm also curious to know how many webhook items get sent per webhook frame for all those affected. You can find this in your log file.
From config.ini:
```# Webhook settings
##################
webhook: http://127.0.0.1:4000
wh-types:[raid]
wh-threads: 1
```
Everything else regarding webhook is set to default. I used to run RM with PokeAlarm way back when hash wasn't yet necessary and everything worked fine back then (like 3 or so months ago). If you need any logs, please specify how can I make the needed ones and I'll post them here.
Just default config. Think it was introduced since the wh-types commit but im not sure.

Btw, could it have something to do with stuff described here?

https://github.com/RocketMap/RocketMap/pull/2248
@darkelement1987 thanks for the heads up! I implemented what was described in that fix to my map, haven't seen any delays for the last three hours. I'll report back in a few more hours and say if I'm gonna stumble upon any more delays.
Nice to hear, just implemented it myself too. I'll report back tomorrow :)

Edit: Still have small delay, last raid i noticed it was 1 min late, not sure what the intended delay is.
Edit 2: Most delayed raid today (out of 10 raids) was 9 minutes late, the least delayed was within 1 min.
Edit 3: Most delayed: 11 min

@zenoncrow ?
@darkelement1987 there were 4 more raids for me yesterday, all were sent on time except one which was off by 30 minutes, but the thing is it might have been a problem with either an account or my internet connection at home where I keep the map, because it didn't even appear on the map itself until it had 1h:30m left, and then raids were off for the day. I'll see if I'm gonna get any delays today.
@sebastienvercammen 

> I'm also curious to know how many webhook items get sent per webhook frame for all those affected. You can find this in your log file.

Exactly where can i find this?
So, it's been working for almost half a day now, no delays for me. Commit #2248 does fix the problem.
then it might be my wh threads that get behind. Ill try increasing them
I want to give you my settings too. I hope you can fix it soon because my instances (With just got a few rare pokemon to be sent)are affected pretty much.
My Opinion: The Webhook is getting delayed because it's having always one Pokemon in the pool which isnt processed. when a second pokemon appears, the notification for both is triggered. If the second pokemon is sent out way to late (for example if the pokemon just spawned 2 hours later) both information are sent to webhook then and of course the first one wasnt triggered because the remaining time went over (negative) 2 hours.
My instance ran from 19:45 so there are only 4 Pokemon, but i guess this is enough to show you the error:
pastebin of the configuration:https://pastebin.com/JmeXA7FZ
(confirmed and checked that no personal information is included)
![ss 2017-08-16 at 02 34 47](https://user-images.githubusercontent.com/30536742/29342759-e17da810-822c-11e7-8340-060bdf3ed215.png)
Here you can see the Pokemon "Porygon" was still triggered, because it appeared at 20:48. so remaining time was around 20 minutes. the snorlax(Relaxo) was the "triggering pokemon" for both of the pokemon at that time. the remaining time of it was ~30min in the message of PokeAlarm.

![ss 2017-08-16 at 02 36 16](https://user-images.githubusercontent.com/30536742/29342797-385dcca0-822d-11e7-9744-b6b674171efb.png)
Here you can see the Telegram notification for both pokemon triggered and given at the same time.

As you can see, the "quappo" wasnt triggered because of its remaining time. i took a look at the stand saw that it spawned at around 22:07(a friend confirmed that it was on the map)
it was processed at 0:32 ; -8720 seconds remaining is 2hours and 25 minutes.
I hope i can help with this :)

Should be fixed with #2248. Please update, test and let us know if this fixed it for you.

Resolving patch:
diff --git a/pogom/webhook.py b/pogom/webhook.py
--- a/pogom/webhook.py
+++ b/pogom/webhook.py
@@ -4,6 +4,7 @@
 import logging
 import requests
 import threading
+from queue import Empty
 from cachetools import LFUCache
 from timeit import default_timer
 from .utils import get_async_requests_session
@@ -81,50 +82,52 @@ def wh_updater(args, queue, key_caches):
     while True:
         try:
             # Loop the queue.
-            whtype, message = queue.get()
-
-            frame_message = {
-                'type': whtype,
-                'message': message
-            }
-
-            # Get the proper cache if this type has one.
-            key_cache = None
-
-            if whtype in key_caches:
-                key_cache = key_caches[whtype]
-
-            # Get the unique identifier to check our cache, if it has one.
-            ident = message.get(ident_fields.get(whtype), None)
-
-            # cachetools in Python2.7 isn't thread safe, so we add a lock.
-            with wh_lock:
-                # Only send if identifier isn't already in cache.
-                if ident is None or key_cache is None:
-                    # We don't know what it is, or it doesn't have a cache,
-                    # so let's just log and send as-is.
-                    log.debug(
-                        'Queued webhook item of uncached type: %s.', whtype)
-                    frame_messages.append(frame_message)
-                elif ident not in key_cache:
-                    key_cache[ident] = message
-                    log.debug('Queued %s to webhook: %s.', whtype, ident)
-                    frame_messages.append(frame_message)
-                else:
-                    # Make sure to call key_cache[ident] in all branches so it
-                    # updates the LFU usage count.
-
-                    # If the object has changed in an important way, send new
-                    # data to webhooks.
-                    if __wh_object_changed(whtype, key_cache[ident], message):
+            try:
+                timeout = frame_interval_sec if len(
+                    frame_messages) > 0 else None
+                whtype, message = queue.get(True, timeout)
+            except Empty:
+                pass
+            else:
+                frame_message = {'type': whtype, 'message': message}
+
+                # Get the proper cache if this type has one.
+                key_cache = None
+
+                if whtype in key_caches:
+                    key_cache = key_caches[whtype]
+
+                # Get the unique identifier to check our cache, if it has one.
+                ident = message.get(ident_fields.get(whtype), None)
+
+                # cachetools in Python2.7 isn't thread safe, so we add a lock.
+                with wh_lock:
+                    # Only send if identifier isn't already in cache.
+                    if ident is None or key_cache is None:
+                        # We don't know what it is, or it doesn't have a cache,
+                        # so let's just log and send as-is.
+                        log.debug('Queued webhook item of uncached type: %s.',
+                                  whtype)
+                        frame_messages.append(frame_message)
+                    elif ident not in key_cache:
                         key_cache[ident] = message
+                        log.debug('Queued %s to webhook: %s.', whtype, ident)
                         frame_messages.append(frame_message)
-                        log.debug('Queued updated %s to webhook: %s.',
-                                  whtype, ident)
                     else:
-                        log.debug('Not queuing %s to webhook: %s.',
-                                  whtype, ident)
-
+                        # Make sure to call key_cache[ident] in all branches
+                        # so it updates the LFU usage count.
+                        # If the object has changed in an important way, send
+                        # new data to webhooks.
+                        if __wh_object_changed(whtype, key_cache[ident],
+                                               message):
+                            key_cache[ident] = message
+                            frame_messages.append(frame_message)
+                            log.debug('Queued updated %s to webhook: %s.',
+                                      whtype, ident)
+                        else:
+                            log.debug('Not queuing %s to webhook: %s.', whtype,
+                                      ident)
+                queue.task_done()
             # Store the time when we added the first message instead of the
             # time when we last cleared the messages, so we more accurately
             # measure time spent getting messages from our queue.
@@ -168,13 +171,11 @@ def wh_updater(args, queue, key_caches):
                                     queue.qsize(),
                                     wh_threshold_lifetime)
 
-            queue.task_done()
         except Exception as e:
             log.exception('Exception in wh_updater: %s.', e)
 
 
 # Helpers
-
 # Background handler for completed webhook requests.
 def __wh_completed(sess, resp):
     # Instantly close the response to release the connection back to the pool.


Test patch: