Remove mouse listeners from intended target.

Related #1381 #1403.