The problem with InvalidOperationException was related to the fact that IsSelected was false when checked, but already becomes true until control flow reaches pattern execution. Hence instead of falling back to patterns immediately, we'll wait a bit to give mouse click a chance to be handled.
Made waiting for input idle tolerate already closed window situation
Cleaner solution for #186