I added one in a previous commit, c394d7bfcfed39f0c8241371b39491bd40c9f52b
Basically hubot ignores anything that has more than three words between 'what' and 'to do'.
If it proves to still be a nuisance perhaps counting words between 'to do' and (a question mark | end of line) will help.