Add automatic test for JJB config

There's a test for the config, it makes no sense not to run it on
submit, especially since it's pretty quick

Change-Id: I503e49f7721aa4d6efba9e8ee3e83e6ff2cc3970