I don't like this part of the code. The possible values of X and Y depend on the value of Z, so having constants 3 and 5 here doesn't seem right.

If the phone is perfectly vertical so Z is 0 then x*x + y*y will equal 9.8*9.8.   If the user is holding it at a 45 degree angle, half way between landscape and portrait, then x and y will each be about 7.

But if Z is 9, then x*x + y*y will only be about 18, and neither coordinate will ever be greater than 5.

I had thought that after testing Z, you could just base the orientation on whether X or Y was greater. But I realize now that would cause flickering back and forth if the user held the phone near 45 degrees, so you'll have to be more careful about that.

Maybe you can try converting X and Y to an actual angle with:

  90 - Math.atan2(Y,X) * 180 / Math.PI

Then you can leave the orientation unchanged if the angle is close to 45, 135, etc, but change it otherwise.