Repository: qtile/qtile

Issue:
libqtile.drawer.Drawer doesn't free Pixmap or GC objects, leading to memory leaks.
I've been debugging a memory leak in Xorg.bin I've been observing for a little while while using qtile, and I believe this is the issue.

I noticed this is most prominent in TreeTab, which unconditionally creates new `libqtile.drawer.Drawer()` objects in `libqtile.layout.tree.TreeTab._resize_panel()`, replacing the old ones without any cleanup. Here's the `xrestop` output of me switching between two windows in a tree layout a few hundred times in a minute:

```
xrestop - Display: localhost:0
          Monitoring 12 clients. XErrors: 0
          Pixmaps:  908726K total, Other:      74K total, All:  908800K total

res-base Wins  GCs Fnts Pxms Misc   Pxm mem  Other   Total   PID Identifier    
0a00000     5 2074    9 2074  144   900747K     61K 900808K   ?   qtile
```

As you can see, qtile isn't cleaning up GC or Pxms, because there 2000 of them. Interestingly, resident memory usage as reported by top was very low, around 12mb. What originally prompted me to look at this was Xorg.bin taking up ~1gb of resident after a few days of running.

Repeating my test with the below (disgusting) hack shows that the GC and Pxms columns remain constant after being cleaned up on `libqtile.drawer.Drawer` destruction.

```
diff --git a/libqtile/drawer.py b/libqtile/drawer.py
index dad2837..8619242 100644
--- a/libqtile/drawer.py
+++ b/libqtile/drawer.py
@@ -196,6 +196,10 @@ class Drawer:
         self.ctx = self.new_ctx()
         self.clear((0, 0, 1))

+    def __del__(self):
+        self.qtile.conn.conn.core.FreePixmap(self.pixmap)
+        self.qtile.conn.conn.core.FreeGC(self.gc)
+
     def _rounded_rect(self, x, y, width, height, linewidth):
         aspect = 1.0
         corner_radius = height / 10.0
```

`xrestop` output with gross hack in place:

```
xrestop - Display: localhost:0
          Monitoring 10 clients. XErrors: 0
          Pixmaps:    1965K total, Other:      20K total, All:    1985K total

res-base Wins  GCs Fnts Pxms Misc   Pxm mem  Other   Total   PID Identifier    
0a00000     5   12    9   12  144     1844K     12K   1857K   ?   qtile
```

Using `__del__` for cleanup is obviously not the correct fix, but it does show this to be the source of those leaking GC and Pixmap objects. Getting the parent objects to do the cleanup would be the best bet I think, but I don't know how their cleanup is performed either - this is my first real dive into the qtile code, but it doesn't look like there's any explicit cleanup of the relevant objects, so it might be a tricky fix.

For reference, I'm running qtile 0.8.0 on Arch Linux, installed via AUR.

If you have a proposed/preferred solution, let me know and I'd be happy to do the work. I'm going to run with the above hack in place for a few days and see if my resident memory usage stays at a normal level, which should tell me if this is the source of the original memory leak I started hunting.



Discussion:
On Sat, Sep 27, 2014 at 04:47:51AM -0700, Nathan Hoad wrote:

> I've been debugging a memory leak in Xorg.bin I've been observing for a little while while using qtile, and I believe this is the issue.
> 
> I noticed this is most prominent in TreeTab, which unconditionally creates new `libqtile.drawer.Drawer()` objects in `libqtile.layout.tree.TreeTab._resize_panel()`, replacing the old ones without any cleanup. Here's the `xrestop` output of me switching between two windows in a tree layout a few hundred times in a minute:
> 
> ```
> xrestop - Display: localhost:0
>           Monitoring 12 clients. XErrors: 0
>           Pixmaps:  908726K total, Other:      74K total, All:  908800K total
> 
> res-base Wins  GCs Fnts Pxms Misc   Pxm mem  Other   Total   PID Identifier    
> 0a00000     5 2074    9 2074  144   900747K     61K 900808K   ?   qtile
> ```
> 
> As you can see, qtile isn't cleaning up GC or Pxms, because there 2000 of them. Interestingly, resident memory usage as reported by top was very low, around 12mb. What originally prompted me to look at this was Xorg.bin taking up ~1gb of resident after a few days of running.
> 
> Repeating my test with the below (disgusting) hack shows that the GC and Pxms columns remain constant after being cleaned up on `libqtile.drawer.Drawer` destruction.
> 
> ```
> diff --git a/libqtile/drawer.py b/libqtile/drawer.py
> index dad2837..8619242 100644
> --- a/libqtile/drawer.py
> +++ b/libqtile/drawer.py
> @@ -196,6 +196,10 @@ class Drawer:
>          self.ctx = self.new_ctx()
>          self.clear((0, 0, 1))
> 
> +    def __del__(self):
> +        self.qtile.conn.conn.core.FreePixmap(self.pixmap)
> +        self.qtile.conn.conn.core.FreeGC(self.gc)
> +
>      def _rounded_rect(self, x, y, width, height, linewidth):
>          aspect = 1.0
>          corner_radius = height / 10.0
> ```
> 
> `xrestop` output with gross hack in place:
> 
> ```
> xrestop - Display: localhost:0
>           Monitoring 10 clients. XErrors: 0
>           Pixmaps:    1965K total, Other:      20K total, All:    1985K total
> 
> res-base Wins  GCs Fnts Pxms Misc   Pxm mem  Other   Total   PID Identifier    
> 0a00000     5   12    9   12  144     1844K     12K   1857K   ?   qtile
> ```
> 
> Using `__del__` for cleanup is obviously not the correct fix, but it does show this to be the source of those leaking GC and Pixmap objects. Getting the parent objects to do the cleanup would be the best bet I think, but I don't know how their cleanup is performed either - this is my first real dive into the qtile code, but it doesn't look like there's any explicit cleanup of the relevant objects, so it might be a tricky fix.
> 
> For reference, I'm running qtile 0.8.0 on Arch Linux, installed via AUR.
> 
> If you have a proposed/preferred solution, let me know and I'd be happy to do the work. I'm going to run with the above hack in place for a few days and see if my resident memory usage stays at a normal level, which should tell me if this is the source of the original memory leak I started hunting.

Thanks for the report! It looks like we probably need to do this
anyway (and it seems fine, although **del** isn't necessarily called
for each python object, I don't have a better suggestion). Let me know
if this fixes your memory leak or if there are other things that need
tracked down.

Thanks!

> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/qtile/qtile/issues/520

So I've been thinking about this a bit more, and looking through our code. It looks like the /only/ place that creates a Drawer that doesn't live for qtile's lifetime is treetab. So I think the best option is probably to switch treetab to only use one drawer (like the rest of qtile).

That said, I think it is probably a good idea to at least attempt to clean up after ourselves, so the `__del__` hack, while not ideal, seems reasonable enough to merge as well. If you want to PR that (and maybe re-arrange treetab to not be so wasteful) that would be great!


Resolving patch:
diff --git a/libqtile/drawer.py b/libqtile/drawer.py
--- a/libqtile/drawer.py
+++ b/libqtile/drawer.py
@@ -194,6 +194,10 @@ def __init__(self, qtile, wid, width, height):
         self.ctx = self.new_ctx()
         self.clear((0, 0, 1))
 
+    def __del__(self):
+        self.qtile.conn.conn.core.FreeGC(self.gc)
+        self.qtile.conn.conn.core.FreePixmap(self.pixmap)
+
     def _rounded_rect(self, x, y, width, height, linewidth):
         aspect = 1.0
         corner_radius = height / 10.0
diff --git a/libqtile/layout/tree.py b/libqtile/layout/tree.py
--- a/libqtile/layout/tree.py
+++ b/libqtile/layout/tree.py
@@ -262,6 +262,7 @@ def __init__(self, **config):
         self.add_defaults(TreeTab.defaults)
         self._focused = None
         self._panel = None
+        self._drawer = None
         self._tree = Root(self.sections)
         self._nodes = {}
 
@@ -545,12 +546,13 @@ def cmd_decrease_ratio(self):
         self.group.layoutAll()
 
     def _create_drawer(self):
-        self._drawer = drawer.Drawer(
-            self.group.qtile,
-            self._panel.window.wid,
-            self.panel_width,
-            self.group.screen.dheight
-        )
+        if self._drawer is None:
+            self._drawer = drawer.Drawer(
+                self.group.qtile,
+                self._panel.window.wid,
+                self.panel_width,
+                self.group.screen.dheight
+            )
         self._drawer.clear(self.bg_color)
         self._layout = self._drawer.textlayout(
             "",


Test patch: