Repository: qutebrowser/qutebrowser

Issue:
Keeping StyleSheetObservers around which could be deleted?
From #1476:

---

@jgkamat:

---

@The-Compiler, this is probably a poor place to report this, but [I found a refloop](https://github.com/qutebrowser/qutebrowser/blob/master/qutebrowser/config/config.py#L503) in `StyleSheetObserver` (which was the #2 *count* of objects on my long-running instance). Because it sets `obj` as a parent and stores `obj` in it's object variables, no `StyleSheetObserver` objects can ever get freed unless it's parent is also freed. Is there a reason why it's being set as a child? If I remove the `parent=obj` in the referenced line, the objects get freed properly (but stylesheets don't update properly anymore?).

This probably isn't going to be a big help though, since python objects seem to take <20mb total anyway.

![objgraph-4_5jqft6](https://user-images.githubusercontent.com/4349709/32594522-5da21858-c4fa-11e7-9ec2-7405a96f670c.png)

---

@The-Compiler:

---

Hmm, interesting! What do you use to generate those graphs?

I thought Python's [garbage collection](https://www.quora.com/How-does-garbage-collection-in-Python-work-What-are-the-pros-and-cons) was here for exactly this situation, when simple reference counting can't detect those cycles... Maybe it just fails when mixing the Qt and Python worlds :wink:

The reason the observer has the object as parent is so the observer gets kept around (by PyQt/Qt) for as long as the object is alive - if you remove that and the style sheets don't update correctly, it's probably because the observer was deleted immediately after being created and getting out of scope.

Does the situation improve if you remove the `_obj` reference, and use `self.parent()` instead?

---

@jgkamat:

---

>Hmm, interesting! What do you use to generate those graphs?

I've been using `objgraph` to generate everything. ([documentation](https://mg.pov.lt/objgraph/)). I used `objgraph.show_most_common_types() ` to show the total count of objects, `objgraph.by_type` to find objects of a type, and `objgraph.show_refs` to generate graphs from ab object.

> I thought Python's garbage collection was here for exactly this situation

I don't think that detecting cycles in this case helps much, the parent `URLText` object is a long-living object. If `URLText` ever becomes safe to delete, then everything will be deleted, but because the `URLText` object lives for the span of the program, all of it's children do too. The problem would be the same as if we were adding `StyleSheetObserver` objects to a list in `URLText` and never deleting them (I think my previous comment about refloop is misleading).

> The reason the observer has the object as parent is so the observer gets kept around (by PyQt/Qt) for as long as the object is alive

Ah, that makes a lot more sense now. However, I later realized stylesheet updating wasn't working for me at all on master. Maybe it only works with webkit (which I can't test now). 

> Does the situation improve if you remove the _obj reference, and use self.parent() instead?

No, it didn't help because the problem would still exist even if it wasn't a refloop, the parent would just keep a ton of transient objects. Since we only connect the signals when `update=True`, can't we move `update` to the constructor and use `self.setParent` when that's true? I found that ~5 StyleSheetObserver objects are created for every page load, with `update=False`. I put some prints and it seems to call the method the same amount of times either way. [Here's a quick patch which adds memory debugging printouts and has the change I'm looking at](http://paste.debian.net/994880/).


Discussion:
> Ah, that makes a lot more sense now. However, I later realized stylesheet updating wasn't working for me at all on master. Maybe it only works with webkit (which I can't test now).

Note this isn't about user stylesheets. It's about Qt stylesheets used to style widgets. If you can do `:set colors.statusbar.normal.bg blue` and your statusbar gets blue, it's working. If you remove the `parent=obj`, that's going to break.

> Since we only connect the signals when `update=True`, can't we move `update` to the constructor and use `self.setParent` when that's true?

That sounds like a good idea. 
In order to actually get some numbers on this, I ran the worlds worst memory profile (using `free` and the `pympler` class tracker). I first made some minimal modifications to make turning on/off the memory patch easy. Then I added a bit of code to create stylesheets at startup (to simulate a long-running process that has created a bunch). It looks like:

```
        from pympler import classtracker
        import gc
        tr = classtracker.ClassTracker()
        tr.track_class(config.StyleSheetObserver)
        tr.create_snapshot()
        # ~6 per page visit, 10000 per qb instance
        # for i in range(10000 * 6):
        #     config.set_register_stylesheet(self, update=False)
        tr.create_snapshot()
        tr.stats.print_summary()
        gc.collect
``` 

Guessing `10000` as a rough number of pages visited/refreshes (I've gone higher than this, but I probably use qb a lot more than most...) in a long running session, multiplied for 6, approx the number of calls per refresh. I then ran a baseline, without patch, and with patch run:

```
*[j@laythe qutebrowser]$ python3 -m qutebrowser --temp-basedir --no-err-windows & sleep 20 && free -b # baseline
[python3] 28314
22:08:39 WARNING: /usr/local/lib/python3.5/dist-packages/pympler/asizeof.py:217: DeprecationWarning: unicode_internal codec has been deprecated
  u = u.encode('unicode-internal')  # see .../Lib/test/test_sys.py

---- SUMMARY ------------------------------------------------------------------
                                         active      0     B      average   pct
                                         active      0     B      average   pct
-------------------------------------------------------------------------------
              total        used        free      shared  buff/cache   available
Mem:    16744714240  2147102720  3230027776   430321664 11367583744 13815431168
Swap:             0           0           0
*[j@laythe qutebrowser]$ free -b
              total        used        free      shared  buff/cache   available
Mem:    16744714240  2026049536  3377512448   405172224 11341152256 13961637888
Swap:             0           0           0
*[j@laythe qutebrowser]$ python3 -m qutebrowser --temp-basedir --no-err-windows # 10000 * 6 ss no patch
22:09:21 WARNING: /usr/local/lib/python3.5/dist-packages/pympler/asizeof.py:217: DeprecationWarning: unicode_internal codec has been deprecated
  u = u.encode('unicode-internal')  # see .../Lib/test/test_sys.py

---- SUMMARY ------------------------------------------------------------------
                                         active      0     B      average   pct
  qutebrowser.config.config.Styl...           0      0     B      0     B    0%
                                         active      0     B      average   pct
  qutebrowser.config.config.Styl...       60000     13.74 MB    240     B    0%
-------------------------------------------------------------------------------*[j@laythe qutebrowser]$ 
              total        used        free      shared  buff/cache   available
Mem:    16744714240  2354216960  3018121216   430968832 11372376064 13607669760
Swap:             0           0           0
*[j@laythe qutebrowser]$ free -b
              total        used        free      shared  buff/cache   available
Mem:    16744714240  2026532864  3371200512   406892544 11346980864 13959434240
Swap:             0           0           0
*[j@laythe qutebrowser]$ python3 -m qutebrowser --temp-basedir --no-err-windows # 10000 * 6 ss with patch
22:10:34 WARNING: /usr/local/lib/python3.5/dist-packages/pympler/asizeof.py:217: DeprecationWarning: unicode_internal codec has been deprecated
  u = u.encode('unicode-internal')  # see .../Lib/test/test_sys.py

---- SUMMARY ------------------------------------------------------------------
                                         active      0     B      average   pct
  qutebrowser.config.config.Styl...           0      0     B      0     B    0%
                                         active      0     B      average   pct
  qutebrowser.config.config.Styl...           0     16     B      0     B    0%
-------------------------------------------------------------------------------
              total        used        free      shared  buff/cache   available
Mem:    16744714240  2282840064  3082326016   434028544 11379548160 13675991040
Swap:             0           0           0
*[j@laythe qutebrowser]$ free -b
              total        used        free      shared  buff/cache   available
Mem:    16744714240  2026119168  3367526400   406880256 11351068672 13959860224
Swap:             0           0           0
*[j@laythe qutebrowser]$ (- 2354216960 2282840064)
71376896
*[j@laythe qutebrowser]$ 

```

What's interesting is that pympler says that 13mb was wasted, but free says 71mb was saved. Even more interestingly, even the patched version is above the baseline, suggesting there's another leak somewhere as well (it might also just be pympler debug data filling up ram as well, so in that case, free is completely off). Either way, I think that this is definetly an improvement (but I'm not sure by how much) so I'll submit the patch after cleaning it up.


Resolving patch:
diff --git a/qutebrowser/config/config.py b/qutebrowser/config/config.py
--- a/qutebrowser/config/config.py
+++ b/qutebrowser/config/config.py
@@ -483,8 +483,8 @@ def set_register_stylesheet(obj, *, stylesheet=None, update=True):
         stylesheet: The stylesheet to use.
         update: Whether to update the stylesheet on config changes.
     """
-    observer = StyleSheetObserver(obj, stylesheet=stylesheet)
-    observer.register(update=update)
+    observer = StyleSheetObserver(obj, stylesheet, update)
+    observer.register()
 
 
 @functools.lru_cache()
@@ -504,9 +504,14 @@ class StyleSheetObserver(QObject):
         _stylesheet: The stylesheet template to use.
     """
 
-    def __init__(self, obj, stylesheet):
-        super().__init__(parent=obj)
+    def __init__(self, obj, stylesheet, update):
+        super().__init__()
         self._obj = obj
+        self._update = update
+
+        # We only need to hang around if we are asked to update.
+        if self._update:
+            self.setParent(self._obj)
         if stylesheet is None:
             self._stylesheet = obj.STYLESHEET
         else:
@@ -525,7 +530,7 @@ def _update_stylesheet(self):
         """Update the stylesheet for obj."""
         self._obj.setStyleSheet(self._get_stylesheet())
 
-    def register(self, update):
+    def register(self):
         """Do a first update and listen for more.
 
         Args:
@@ -535,5 +540,5 @@ def register(self, update):
         log.config.vdebug("stylesheet for {}: {}".format(
             self._obj.__class__.__name__, qss))
         self._obj.setStyleSheet(qss)
-        if update:
+        if self._update:
             instance.changed.connect(self._update_stylesheet)


Test patch:
diff --git a/tests/unit/config/test_config.py b/tests/unit/config/test_config.py
--- a/tests/unit/config/test_config.py
+++ b/tests/unit/config/test_config.py
@@ -601,7 +601,7 @@ def setStyleSheet(self, stylesheet):
 def test_get_stylesheet(config_stub):
     config_stub.val.colors.hints.fg = 'magenta'
     observer = config.StyleSheetObserver(
-        StyleObj(), stylesheet="{{ conf.colors.hints.fg }}")
+        StyleObj(), stylesheet="{{ conf.colors.hints.fg }}", update=False)
     assert observer._get_stylesheet() == 'magenta'