Only drag numbers when over the draggable area.

This fixes #365.