Improve some sceAtrac functions

It's still not how it really works, but at least it fixes some bugs in previous commits.
The fix in #1797 broke some games which works normally before such as Fate Unlimited Code, God of War and game in #1815. Based on some tests by @raven02, games which fixed by #1797 only needs return a <-1 value such as -2 or even -0x100. So I make another fix for that. Based on some tests, it should not break other games.

The fix in #1805 removes a hack way for numSamples. However, it is probably still necessary for some games, because some other functions are still not working correctly. And then I make a better hack way for that. It has no affect for those games working normally, but it would not crash some games which still don't work normally right now.