Better error handling for bad activity duplicate

This either fixes [#61488076] or provides a better experience if it's not fixed.

https://www.pivotaltracker.com/story/show/61488076