Yes, the files must be created before running the test.
The .pem file asks for a lot of input, I haven't seen yet if it can be run without interaction.
If so, of course we could autogenerate these files.