twoolie/NBT — Issue #26: Store TAG_Byte_Array as a bytearray instead of a string
Author: macfreek
Created: 2012-03-08 20:09:03+00:00
State: open

The current nbt module stores the value of a TAG_Byte_Array as string. This has a few problems. It is not mutable, it is slow, and it may give problems when converting to Python 3 (since all strings are Unicode in Python 3).

I propose to store it as a bytearray, which is a native Python object for, well, a byte array. This is different from a byte object in that it is mutable.

The code change in nbt.py itself is rather simple, see 0a700.

However, it also requires changes in code that assumes it is a string, such as chunk.py. See c0aa8. It
Fixing chunk.py is not bad, but this may also break other code that relies on this internal structure of TAG_Byte_Array.

What do you think? I'm very much in favour of changing it for speed, Python 3 compatibility and easy-of-use.

Let me demonstrate the easy of use with an example. This is the current code required to modify an item in the HeightMap:

heightBytes = mynbt['Level']['HeightMap'].value
heightdata = list(struct.unpack("32768B", heightBytes))
heightdata[3] = 63
heightBytes = array.array('B', heightdata).tostring()
mynbt['Level']['HeightMap'].value = heightBytes

If the value parameter is mutable, it could greatly simplifies the above code. There is no need to pack or unpack anymore:

mynbt['Level']['HeightMap'].value[3] = 63

In fact, it is even trivial to add a __getitem__ and __setitem__ function to TAG_Byte_Array (see e8cd3), so one can simply write:

mynbt['Level']['HeightMap'][3] = 63

In my opinion, this is MUCH cleaner than the current code.

My question to all users of NBT is: is the above advantage worth the downside that it will likely break code that still expects ```TAG_Byte_Array().value``` to be string? ```struct.unpack('256b', my_tag_byte_array.value)``` will raise a ```struct.error``` , although ```array.array('B', my_tag_byte_array.value)``` will continue to work just fine (although setting my_tag_byte_array.value to a string using ```array.array().tostring()``` may fail things in nbt.py)

As an added bonus, it is a bit faster too (it is actually 600 times faster, but that's negated by the slow method to parse the NBT). Here are some timing measurements:

current code: unpack (returns a tuple) and convert to list

mynbt = nbt.NBTFile(filename) 2500 µs excluding actual I/O
blocksBytes = mynbt['Level']['Blocks'].value 6.7 µs
blocktuple = struct.unpack("32768B", blocksBytes) 350 µs
blockdata = list(blocktuple) 180 µs

list: a slow alternative

mynbt = nbt.NBTFile(filename) 2500 µs excluding actual I/O
blocksBytes = mynbt['Level']['Blocks'].value 6.7 µs
blockdata = [i for i in blocksBytes] 2300 µs

bytes: an immutable byte sequence. blazingly fast.

mynbt = nbt.NBTFile(filename) 2500 µs excluding actual I/O
blocksBytes = mynbt['Level']['Blocks'].value 6.7 µs
blockdata = bytes(blocksBytes) 0.2 µs

bytearray: my proposal

mynbt = nbt.NBTFile(filename) 2500 µs excluding actual I/O
blocksBytes = mynbt['Level']['Blocks'].value 6.7 µs
blockdata = bytearray(blocksBytes) 4.2 µs

array: an equally fast alternative.

mynbt = nbt.NBTFile(filename) 2500 µs excluding actual I/O
blocksBytes = mynbt['Level']['Blocks'].value 6.7 µs
blockdata = array.array('B', blocksBytes) 4.4 µs