Return SCE_KERNEL_ERROR_ILLEGAL_PRIORITY when priority < 0

It didn't fix anything yet but i think somehow related to issue #3759