Make is possible to click a detached button again (#12781)

Partial revert of the earlier fix

Change-Id: I67d016f1ea073189fdb53fc99f7c1b45b21ac124