Repository: evennia/evennia

Issue:
Bug: Lag with newest change of Tag/Aliashandler
#### Brief summary of issue / Description of requested feature:
After updating with the latest changes, the game immediately began to have crippling lag issues where anything with a sizable number of aliases/tags were concerned. For example, a 'look' command might take 20-30 seconds or the like. Speculation is this is due to the sheer number of tags that were in my database, which is in the tens of thousands range. Watching in `htop` would see the CPU usage for the server process continually spiking to 100%, and commands would print out lines from their return with noticeably delays between each line. (such as an `evtable` taking a quarter of a second between each line of the table displaying).

#### Steps to reproduce the issue / Reasons for adding feature:

1. Compare 'look' or other commands that reference tags to others that don't

#### Error output / Expected result of feature
Commands where they check tags/aliases are vastly slower for no apparent reason.


#### Extra information, such as Evennia revision/repo/branch, operating system and ideas for how to solve / implement:
Latest change to the tags/aliashandler. SQLite database. 17746 tags in the database. Test room had around 80 or so aliases between the different objects.

Additional note: On my test server, I initially saw this behavior for around a half hour, and then it just spontaneously stopped. So it might be related to loading the AliasHandler/TagsHandler, possibly? I had 13 other players on my live server when I saw this behavior, so it could have just been enough people put enough strain on it that the situation continued indefinitely, but might have eventually resolved on its own.

Another edit: I could partially reproduce this on my test environment when doing a `@tag/search <tagname>` that would return a few hundred matches and then immediately doing a `look` - I'd see a noticeable slowdown for several seconds. It seemed to resolve itself immediately after. Maybe the login process for characters is producing something similar, but with a much greater duration?


Discussion:
My current running theory is that for some reason, the SQL query constructed by the caching in the attributehandler and/or tagshandler is markedly different in performance in my live game than in other environments. Looking at the query that's done to construct the cache, is there a reason why it goes to the through model and then refers back to the tag/attribute, rather than just using the many-to-many relationship directly? For example, https://github.com/evennia/evennia/blob/master/evennia/typeclasses/attributes.py#L280 - we have:
```
query = {"%s__id" % self._model : self._objid,
                         "attribute__db_model" : self._model,
                         "attribute__db_attrtype" : self._attrtype,
                         "attribute__db_category__iexact" : category.lower() if category else None}
                attrs = [conn.attribute for conn in getattr(self.obj,
                            self._m2m_fieldname).through.objects.filter(**query)]
```
Could this be rewritten as:
```
query = {"db_model" : self._model,
               "db_attrtype" : self._attrtype,
               "db_category__iexact" : category.lower() if category else None}
                attrs = getattr(self.obj, self._m2m_fieldname).filter(**query)
```
I'm not sure if the queries would have equivalent SQL - I might try to print out what the .query is for the querysets to check.
As an additional note, something that occurred to me now - a month or so ago I ran into a mystifying bug for a queryset I was building in a particular command. If I executed it at the start of the command, the query took a really long time - say 10 seconds. Moving the query later in the command made it run in a fraction of that - nothing that would make it cached/evaluated before that point. Printed out the .query, same SQL in both cases. That was completely baffling to me, and I wonder if there might be something like that going on here.
I looked a little bit at the profile myself with snakeviz, and I'm new to profiling stuff, but I noticed that 164 seconds out of the 168 of the cumulative time from stuff that seems related to commands was spent in `attributes.py _get_cache` and the resultant `query.py`'s call to `compiler.execute_sql`, which suggests to me that most of the runtime was just devoted purely to the query from _get_cache. I wonder if the execute_sql might vary based on the site packages that are installed or the platform -  whether it builds entirely different sql queries for some reasons, or they're run differently in some way?
@TehomCD As for the two different SQL queries: The `db_tags` field on the ObjectDB is a many-to-many relationship. Django handles this by automatically creating a hidden relation table named `objects_objectdb_db_tags` that looks like this: 

| id  |   objectdb_id  |  tag_id | 
|----|-------------------|----------|
|  1  |              23     |      34   |
|  2 |               12     |      10   |

This connection table is accessed with `object.db_tags.through` and can be queried like any other object. The SQL query used will query this to get the m2m relation for the tags matching the criteria given by id, db_model, db_tagtype, db_key and db_category, one the takes the tag part from that connection. Your alternative goes the other way around, by fetching the Tags by locating this m2m relation and from one step off so-to-say find the tags we need. 

Looking at the SQL they are of course different, but neither seem to have a speed advantage. Yours is however easier to read. Might be worth testing in your particular environment to see if it makes a difference. 

The thing is that the *only* thing that changed (or at least was intended to change) with that latest  update was that Evennia started to actually honor the already existing `db_model` field. 

This makes me wonder - I vaguely remember you having to add `db_model` manually when you updated Arx to the latest Evennia? _Did you remember to put an index on the db_model field?_ You can check it from your game dir like this: 

    > evennia dbshell
    $sqlite> explain query plan select * from typeclasses_tag where "db_model"="objectdb";
    0|0|0|SEARCH TABLE typeclasses_tag USING INDEX typeclasses_tag_36df3c33 (db_model=?)

If you don't see USING INDEX you have probably explained the slowdown - without an index the database must loop the entirety of the your tag table to find the `db_model` you want. 

Edit: Do the same check for Attributes, obviously.
I checked typeclasses_tag and typeclasses_attribute in dbshell, and both reported USING INDEX. However, I was surprised when I tried to use dbshell and it told me sqlite3 wasn't installed, and I had to do so before I could use the command. Could that have made any difference, or is that only for using the dbshell itself?
Hm, that would have been the simple solution. 

Here are some more things to try:
- Start a vanilla Evennia game (vanilla game dir) and populate it with 10000 objects + 50 000 tags or something like that. When running on the same server, does it slow down (it does not for me). 
- Considering the history of Arx jumping Evennia versions, examine the database schema of the vanilla Evennia database and examine it side by side with the Arx one. Make sure all index tables are there etc (some are used specifically for unique-relations for example). The "SQLite Manager" Firefox plugin is good for a quick graphical overview.
- Start a new instance of Arx (using a copy of your game dir) with an empty database, add a lot of objects and tags as above, see if you see a slowdown then. 
When comparing the database schemas, make sure to run the latest revisions of both Arx and the vanilla one since a unique-together-index was added just in the last few days.
... And no, `sqlite` is just the client program for viewing the database, it has no impact on performance.
I did a little experiment using https://docs.djangoproject.com/en/1.10/faq/models/#faq-see-raw-sql-queries . On my dev and live server, I turned `debug = True`, then I had my character object go to the same room, executed `@py from django.db import reset_queries;reset_queries()` and `@py from django.db import connection;print connection.queries` just to see what the differences would be. The pastebins, though largely inscrutable, are http://pastebin.com/Y25038iD and http://pastebin.com/EjLbhjgx . It seems to show that for the same queries for attributes, they run in 0.000 time on the dev server, while on live they run between 0.1 and 0.3 or so - which collectively is enormous. Could there be something that makes it act like it's not indexed, even though dbshell said it was?
Just as a note for myself when mocking the database, Arx has around 160k Attributes. Commenting out the db_model field from the query for Attributes did instantly resolve the problem, but there's still no reasonable explanation as to why, since it should be indexed.
I finally was able to reproduce this on my test environment. After changing to WAL mode on my live environment, I upgraded to the latest version of sqlite on test environment so that I'd be able to open the database (the version which ships with python on windows is too old to open WAL databases), and then immediately saw the lag issue appear when I switched to master branch. So for some reason, going from 3.6 in sqlite to 3.11.0 or later (3.16.0 is the version I upgraded to for my test environment) instantly caused the issue to appear. The error itself still makes no sense at all, but at least now I was able to reproduce it.
Oh as a quick additional note, I had to make sure to check the sqlite version with python shell by doing
```
>>> import sqlite3
>>> sqlite.sqlite_version
```
The version that will display when opening `evennia dbshell` isn't necessarily what python will use, apparently - I thought I had an up to date version of sqlite, but apparently not until I replaced the `sqlite3.dll` in my python/dll directory.
That is very interesting. Is there a reason you are using WAL mode specifically?
I use sqlte version 3.8 here, will try to see if I can update to see if I can reproduce with this new info. 
I decided to go with WAL mode because of supporting slightly greater concurrency, from what I read - since reads don't block writes and vice-versa, hopefully it'd make a long operation on a webpage which are mostly reads not blocking someone attempting to do a write in-game. Since the database locks appeared to be all based on threads from the webpage, hopefully that'll make the problem of locking far more rare, if not vanish entirely. Anecdotally people seem to think WAL mode runs slightly faster than the default (the sqlite page seems to suggest it will in some circumstances), but that could just be a placebo.
Ok, so WAL mode is something later added, it can't in itself be the cause of the lag then.
Yeah, just to be absolutely sure it was unrelated, I switched back and forth from WAL mode to DELETE, and it didn't affect whether the lag appeared - just whether or not I had the `db_model` part of `AttributeHandler`'s queries enabled.
Hm, the latest official precompiled sqlite3 from their website is 3.8.2 for Linux. Did you compile from sources then?
Is 3.16 considered stable?
Oh, never mind. I can apparently not run the correct programs from the command line. :) 3.16 it is. 
@TehomCD So I changed my sqlite3 to use 3.16. I then loaded my "Arx-like" example database which has 70 000 tags/aliases associated with 10 000 objects (4 aliases, 4 tags per object - some tags were deliberately shared between many objects for testing). And alas I can do `look #6999` without any noticeable lag. All commands I try work fast and efficiently, even `@objects`. Could you give me some more examples to test?
Hm, I might still be using sqlite 3.8 actually; importing sqlite from in-game seems to suggest so. I might have to do the same thing you did, look for a library somewhere rather than the sqlite binary. 
I just downloaded the precompiled zip from their download page and copied the sqlite3.dll file over the one python was using - I'm not sure if there's an easier method for getting it set up.
It seems all Linux distros still use 3.8, not sure I want to manually replace the core library like that on my running machine; sqlite is used by many things, it sounds like a dangerous thing to do. :/ 
Yeah, I don't blame you. I actually have no recollection of how I installed the version of sqlite I have on the live machine at all. Maybe linode just had it pre-installed for me.
I'll hold off on it for the moment, but if we have finally narrowed down the cause of this, then this must be down to a difference either in the performance of the later sqlite (highly unlikely) or due to some PRAGMA default changing between the old and the new (much more likely). Maybe the optimal performance is no longer to be found in the settings we use for sqlite. You can see which PRAGMAS we set [here](https://github.com/evennia/evennia/blob/master/evennia/server/server.py#L181), maybe that's worth experimenting with. 
Playing around with this a little more, the db_model query doesn't cause any lag when it's used in conjunction with db_key, only with db_attrtype and/or db_category. No clue why yet, and a month or so ago I should note here that you did a brief test after upgrading to the newest version of sqlite and were unable to reproduce the lag, so it remains a mystery.
I read a small note somewhere that mentioned weirdness about LIKE statements not using indexes, and purely as an experiment, added __iexact to the db_model query. Lag vanished instantly. I don't know why, but well, I guess that's an easy way to solve the bug at last.

Resolving patch:
diff --git a/evennia/typeclasses/attributes.py b/evennia/typeclasses/attributes.py
--- a/evennia/typeclasses/attributes.py
+++ b/evennia/typeclasses/attributes.py
@@ -219,7 +219,7 @@ def __init__(self, obj):
     def _fullcache(self):
         """Cache all attributes of this object"""
         query = {"%s__id" % self._model: self._objid,
-                 "attribute__db_model": self._model,
+                 "attribute__db_model__iexact": self._model,
                  "attribute__db_attrtype": self._attrtype}
         attrs = [conn.attribute for conn in getattr(self.obj, self._m2m_fieldname).through.objects.filter(**query)]
         self._cache = dict(("%s-%s" % (to_str(attr.db_key).lower(),
@@ -273,7 +273,7 @@ def _getcache(self, key=None, category=None):
                     return []  # no such attribute: return an empty list
             else:
                 query = {"%s__id" % self._model: self._objid,
-                         "attribute__db_model": self._model,
+                         "attribute__db_model__iexact": self._model,
                          "attribute__db_attrtype": self._attrtype,
                          "attribute__db_key__iexact": key.lower(),
                          "attribute__db_category__iexact": category.lower() if category else None}
@@ -298,7 +298,7 @@ def _getcache(self, key=None, category=None):
             else:
                 # we have to query to make this category up-date in the cache
                 query = {"%s__id" % self._model: self._objid,
-                         "attribute__db_model": self._model,
+                         "attribute__db_model__iexact": self._model,
                          "attribute__db_attrtype": self._attrtype,
                          "attribute__db_category__iexact": category.lower() if category else None}
                 attrs = [conn.attribute for conn


Test patch: