fix GLib-CRITICAL assertion failures when drag selecting text

The issue was that you got a bunch of these:

```
(aporia:6351): GLib-CRITICAL **: Source ID 18373 was not found when attempting to remove it
```

when drag selecting text. Here is an attempted fix, any comments welcomed.