st/nine: return D3DERR_INVALIDCALL when trying to create a texture of bad format

Signed-off-by: Axel Davy <axel.davy@ens.fr>