Add test to warn when we need to bump cache version

Also add teardown to schema change test to allow tests to be run
afterwards.