Remove mouse listeners from intended target.

Related #1381.