Make sure we handle having a "problems" directory under the application

closes #3184