Fix performance regression with only one active terminal widget.

https://bugzilla.gnome.org/show_bug.cgi?id=721944