scePsmfPlayerConfigPlayer(): Return invalid ERROR code for invalid config mode and config value

Referencing 
https://code.google.com/p/jpcsp/source/browse/trunk/src/jpcsp/HLE/modules150/scePsmfPlayer.java#661

It basically try to return error code for those invalid config mode and config value.

May help some games which uses PSMF.