I'd avoid the cast and the instanceof check. By either:

* declaring byteSize at the top of the method and set in the existing if (limit == null) block and don't run the for loop that increments it when limit == null or
* always use MappedByteBufferSortedQueue, even in the case of a limit, since it bound the memory usage nicely.

I'd probably favor the latter, since we've had some complaints about memory exceptions when the limit is set to be very high. We can perf test with this change and play around with the spooling threshold to see what affect it has.