Be more selective about where Idris tries to make buffers clickable

Hopefully, this will fix the problem @sellout was having on IRC.