Repository: evennia/evennia

Issue:
Duplicate script calls
#### Brief summary of issue / Description of requested feature:
Very rarely, I see the `at_repeat` for a script where there is only a single instance of it being called multiple times, usually with about a minute separation of it. 


#### Steps to reproduce the issue / Reasons for adding feature:

1. It's happened very rarely, and only after long periods of usage without the server being reloaded.

#### Error output / Expected result of feature
`at_repeat` is called multiple times, say twice within a minute or two of the first call.


#### Extra information, such as Evennia revision/repo/branch, operating system and ideas for how to solve / implement:
My guess is that a redundant `LoopingCall` is being created, but how or why is a complete mystery to me. It doesn't happen when the script is first created, because the script had run for days without interruption, and then we start to see the duplicate behavior, seemingly for no reason at all.

As a note, I did see this back in Evennia 0.6 as well, or at least I thought I did, with a different script. I noticed that a value which was supposed to be incremented at a script's `at_repeat` happened far, far more times than should have been possible. Since it was a timer, I just moved to datetime timedelta comparisons and thought no more of it. It was far more visible on the script I just saw because its `at_repeat` does a server broadcast, and I saw duplicate messages.



Discussion:
@TehomCD I take it this is a global script. How long of a period is it using? 
@Griatch Pretty long - 3600 seconds.
Noticed it happening after 2 days of uptime/between `@reload`s for a much shorter script - this one with a 300 second interval. It's pretty hard to notice unless the script does something that makes it obvious, like announcements during at_repeat.
@TehomCD 

I'm running the following script on my local machine: 

```python
import time

class TestScript(Script):
    def at_script_creation(self):
        self.key = "test_repeat_script"
        self.desc = "test #1519"
        self.interval = 10
        self.persistent = True

    def at_start(self):
        self.ndb.started = time.time()
        self.ndb.repeats = 0

    def _get_expected(self):
        return abs(time.time() - self.ndb.started) // self.interval

    def at_server_reload(self):
        print("Reload: TestScript repeats/expected: %s/%s" % (self.ndb.repeats, self._get_expected()))

    def at_server_shutdown(self):
        print("Shutdown: TestScript repeats/expected: %s/%s" % (self.ndb.repeats, self._get_expected()))

    def at_repeat(self):
        if self.ndb.repeats != self._get_expected():
            print("TestScript repeats/expected: %s/%s" % (self.ndb.repeats, self._get_expected()))
        self.ndb.repeats += 1
```

This will reset on a reload which might be a potential problem. But I'll let it run for a while to see if I start seeing discrepancies between expected and actual repeats. You could maybe try to run the same to see if you see an effect. Suggestions for changes to make it more like your real-life case is welcome. 
Since this has only happened after long periods of use, you might try making the cache flush at some point and seeing if that could be a cause also. I'm not sure if that's related, but it might be happening only after a cache flush.

In general, the duplicate at_repeat call was usually about a minute after the first one - not exactly, but around there. Might be worth including timestamps with the print statements so if it pops up you can see the exact time difference.
I'm trying something out to see if I can eliminate the duplicate `ExtendedLoopingCall`s. What I'm doing is overriding `at_idmapper_flush` for my Script typeclass and adding in something to stop the task of a flushed script, adding the current run time/callcount to a global variable dict, since I didn't want to do database writes during a flush. While this killed the tasks, the timer for validating scripts to restart them was up to an hour, so I added a callLater to a class method that calls the ScriptDB manager to validate them. However, without an active `self.ndb._task` the `start()` method wouldn't restart them while their `is_active` attribute was True, so I added a check in `start()` for _task being dead and then looking up in the dict to restart it with the appropriate values, analagous to unpausing. I'll see if this resolves the duplicate scripts, but I'm kinda hopeful.

Resolving patch:
diff --git a/evennia/scripts/scripts.py b/evennia/scripts/scripts.py
--- a/evennia/scripts/scripts.py
+++ b/evennia/scripts/scripts.py
@@ -18,6 +18,17 @@
 __all__ = ["DefaultScript", "DoNothing", "Store"]
 
 
+FLUSHING_INSTANCES = False  # whether we're in the process of flushing scripts from the cache
+SCRIPT_FLUSH_TIMERS = {}  # stores timers for scripts that are currently being flushed
+
+
+def restart_scripts_after_flush():
+    """After instances are flushed, validate scripts so they're not dead for a long period of time"""
+    global FLUSHING_INSTANCES
+    ScriptDB.objects.validate()
+    FLUSHING_INSTANCES = False
+
+
 class ExtendedLoopingCall(LoopingCall):
     """
     LoopingCall that can start at a delay different
@@ -278,6 +289,27 @@ def remaining_repeats(self):
             return max(0, self.db_repeats - task.callcount)
         return None
 
+    def at_idmapper_flush(self):
+        """If we're flushing this object, make sure the LoopingCall is gone too"""
+        ret = super(DefaultScript, self).at_idmapper_flush()
+        if ret and self.ndb._task:
+            try:
+                from twisted.internet import reactor
+                global FLUSHING_INSTANCES
+                # store the current timers for the _task and stop it to avoid duplicates after cache flush
+                paused_time = self.ndb._task.next_call_time()
+                callcount = self.ndb._task.callcount
+                self._stop_task()
+                SCRIPT_FLUSH_TIMERS[self.id] = (paused_time, callcount)
+                # here we ensure that the restart call only happens once, not once per script
+                if not FLUSHING_INSTANCES:
+                    FLUSHING_INSTANCES = True
+                    reactor.callLater(2, restart_scripts_after_flush)
+            except Exception:
+                import traceback
+                traceback.print_exc()
+        return ret
+
     def start(self, force_restart=False):
         """
         Called every time the script is started (for persistent
@@ -294,9 +326,19 @@ def start(self, force_restart=False):
                 started or not. Used in counting.
 
         """
-
         if self.is_active and not force_restart:
-            # script already runs and should not be restarted.
+            # The script is already running, but make sure we have a _task if this is after a cache flush
+            if not self.ndb._task and self.db_interval >= 0:
+                self.ndb._task = ExtendedLoopingCall(self._step_task)
+                try:
+                    start_delay, callcount = SCRIPT_FLUSH_TIMERS[self.id]
+                    del SCRIPT_FLUSH_TIMERS[self.id]
+                    now = False
+                except (KeyError, ValueError, TypeError):
+                    now = not self.db_start_delay
+                    start_delay = None
+                    callcount = 0
+                self.ndb._task.start(self.db_interval, now=now, start_delay=start_delay, count_start=callcount)
             return 0
 
         obj = self.obj


Test patch: