# Python Concepts/Bytes objects and Bytearrays
## Objective

* What is a bytes object?
* Why is a bytes object important?
* What is the difference between a bytes object and a bytearray?
* How is a bytes object created and used?
* How do you convert from a bytes object to other sequences based on bytes?
* How do you avoid errors when using bytes objects or bytearrays?

## Lesson

* One byte is a memory location with a size of 8 bits. A bytes object is an immutable sequence of bytes, conceptually similar to a string.
* Because each byte must fit into 8 bits, each member $x$ of a bytes object is an unsigned int that satisfies $0\leq x\leq 0b1111\_1111.$
* The bytes object is important because data written to disk is written as a stream of bytes, and because integers and strings are sequences of bytes.
* How the sequence of bytes is interpreted or displayed makes it an integer or a string.

## bytes objects

* bytes object displayed
* A bytes object is displayed as a sequence of bytes between quotes and preceded by 'b' or 'B':
* >>> bytes(3) # This initialization produces an empty sequence of 3 bytes.
* b'\x00\x00\x00'
* >>> bytes([3]) # Initialized with a list containing 1 member.
* b'\x03'
* >>> B'\x00\x00\x00'
* b'\x00\x00\x00'
* >>> isinstance(b'\x00\x00\x00', bytes)
* True
* >>> len(b'\x00\x00\x00')
* 3 # bytes
* The representation '\x00' is not read literally. This representation means a byte with value 0x00.
* If a member of a bytes object can be displayed as a printable ASCII character, then it is so displayed.
* >>> B"123ABC"
* b'123ABC'
* >>> B'''\x03 1 2 x y \xE7'''
* b'\x03 1 2 x y \xe7'
* >>> b"""\x41\x42\x43"""
* b'ABC'
* When you look at the contents of a bytes object, it is easy to overlook embedded ASCII characters:
* >>> B'\x05\x97\xa3q\xf9\x17\x83' == b'\x05\x97\xa3' + b'q' + B'\xf9\x17\x83'
* True
* Parts of a bytes object:
* >>> b'0123456789'[5]
* 53 # Individual member is returned as int.
* >>> chr(53)
* '5'
* >>> b'0123456789'[5:8]
* b'567' # Sequence of members is returned as slice.
* >>> b'0123456789'[5:6]
* b'5' # A slice containing 1 byte.
* >>> b'0123456789'[2::3] # A slice containing the byte in position 2, then every 3rd until end.
* b'258'
* >>> b'0123456789'[2::3] == b'0123456789'[2:3] + b'0123456789'[5:6] + b'0123456789'[8:9]
* True
* >>> b1 = b'0123456789'
* >>> b1[7:3:-1]
* b'7654' # slice of b1 reversed
* >>> b1[-1: -len(b1)-1: -1]
* b'9876543210' # b1 reversed
* >>> ( b1[-1: -len(b1)-1: -1]
* ... == b1[-1:: -1]
* ... == b1[len(b1)-1:: -1]
* ... == b1[:: -1] # A simple way to reverse the bytes object.
* ... == b1[len(b1)-1: -len(b1)-1: -1] )
* True # Slicing works as expected.
* Some control characters are recognized as such but not displayed as such:
* >>> b' \v \f \n \t '
* b' \x0b \x0c \n \t '
* bytes object initialized
* >>> B'\x20\x19\x61\x62\x39\x40' # hex values
* b' \x19ab9@'
* >>> b'\101\102\103\104' # octal values
* b'ABCD'
* >>> b'The quick, brown fox ...' # ASCII values
* b'The quick, brown fox ...'
* >>> b'\x54\x68\x65 \161\165\151\143\153, brown \x2E\x2e\056\056' # Mixture.
* b'The quick, brown ....'
* >>> b'\056\056\366\367'
* b'..\xf6\xf7' # If ASCII cannot be displayed, hex is displayed.
* The bytes object can contain recognized control characters:
* >>> a = b'''line1
* ... line2
* ... line3 '''
* >>> a
* b'line1\n line2 \n line3 \t\t\t'
* As for strings prefix 'r' or 'R' may be used:
* >>> b1 = B'\x41\x42' ; b1 ; len(b1)
* b'AB'
* 2
* >>> b2 = rB'\x41\x42' ; b2 ; len(b2)
* b'\\x41\\x42'
* 8
* >>> b2 == b'\x5C' + b'x41' + B"""\134""" + b'x42'
* True
* A suitable sequence can be converted to bytes:
* >>> b1 = bytes([16, 0x20, ord('a'), 98, 0x63, 0o144, ord(' '), 0xEe]) # Input to bytes is list.
* >>> b1
* b'\x10 abcd \xee'
* >>> b2 = bytes( tuple(b1) ) ; b2 # Input to bytes is tuple.
* b'\x10 abcd \xee'
* >>> b2 == b1
* True
* >>> b2 is b1
* False # A deep copy.
* >>> b2 = bytes( [ ord(p) for p in '123 abc DEF' ] ) ; b2 # Converting string to bytes.
* b'123 abc DEF'
* >>> bytes( [ ord(p) for p in 'xyz £ § « ® ½ Ø ñ' ] )
* b'xyz \xa3 \xa7 \xab \xae \xbd \xd8 \xf1' # Characters chr(0x80) through chr(0xFF) are not displayable ASCII.
* >>> bytes([0x123])
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* ValueError: bytes must be in range(0, 256)
* Like a string the bytes object doesn't support item assignment:
* >>> a
* b'line1\n line2 \n line3 \t\t\t'
* >>> a[3]=6
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* TypeError: 'bytes' object does not support item assignment
* Like a string the bytes object can be repeated:
* >>> B"1,2," * 3
* b'1,2,1,2,1,2,'
* Behavior of str and behavior of bytes can be significantly different:
* >>> c1 = '\u0041' ; c1 ; len(c1)
* 'A'
* 1
* >>> b1 = b'\u0041' ; b1 ; len(b1)
* b'\\u0041'
* 6
* The concatenation of 2 or more bytes objects:
* >>> b'abc' + B''' ''' + b"""\x44\x45\x46""" + b"\040" + B'\06123'
* b'abc DEF 123'
* bytes object as iterable
* The bytes object accepts the usual operations over iterables:
* >>> b1 = b'\t\n abcd'
* >>> if 'a' in b1 : print ('found')
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* TypeError: a bytes-like object is required, not 'str'
* >>> if b'a' in b1 : print ('found')
* found
* >>> if b'\n ab' in b1 : print ('found')
* found
* ord('c')
* 99
* >>> if 99 in b1 : print ('found')
* found
* >>> for p in b1 : print (p)
* 9 # p is returned and displayed as int.
* 10
* 32
* 97
* 98
* 99
* 100

## Conversion to bytes object

from int
The following code illustrates the process for a positive integer:
def int_to_bytes (input_int) :
isinstance(input_int, int) or exit (99)
(input_int >= 0) or exit (98)
if (input_int == 0) : return bytes([0])
L1 =
num_bits = input_int.bit_length
while input_int :
L1[0:0] = [(input_int & 0xFF)]
input_int >>= 8
if (num_bits % 8) == 0 :
L1[0:0] = [0]
return bytes(L1)
for i1 in [0, 0x8_9a_bc_de, 0xa8_9a_bc_de] :
b1 = int_to_bytes (i1)
print ('''i1 =, b1 ='''.format(hex(i1), b1))
i1 = 0x0, b1 = b'\x00'
i1 = 0x89abcde, b1 = b'\x08\x9a\xbc\xde'
i1 = 0xa89abcde, b1 = b'\x00\xa8\x9a\xbc\xde'
Method int.to_bytes(length, ....)
Method int.to_bytes(length, byteorder, *, signed=False) returns a bytes object representing an integer.
>>> 0x12_84.to_bytes(2, 'big', signed=True)
b'\x12\x84'
>>> (-0xE2_04).to_bytes(3, 'big', signed=True) # Note the parentheses: (-0xE2_04).
b'\xff\x1d\xfc' # 3 bytes for a signed, negative int.
from str
If each character of the string fits into one byte, the process is simple:
>>> s1 = '\011\012\015\016 123 abc \345\346\347' ; s1
'\t\n\r\x0e 123 abc åæç'
>>> L1 =
>>> for p in s1 : L1 += [ord(p)]
>>> L1
[9, 10, 13, 14, 32, 49, 50, 51, 32, 97, 98, 99, 32, 229, 230, 231]
>>> bytes( L1 )
b'\t\n\r\x0e 123 abc \xe5\xe6\xe7'
A listcomp simplifies the process:
>>> b1 = bytes( [ ord(p) for p in s1 ] ) ; b1
b'\t\n\r\x0e 123 abc \xe5\xe6\xe7'
The above implements encoding 'Latin-1':
>>> b1a = s1.encode('Latin-1') ; b1a
b'\t\n\r\x0e 123 abc \xe5\xe6\xe7'
>>> b1 == b1a
True
Method str.encode creates a bytes object containing the string str encoded:
>>> s1
'\t\n\r\x0e 123 ሴ abc åæç'
>>> s1.encode # Each character 'åæç' occupies one byte but is encoded as 2 bytes.
b'\t\n\r\x0e 123 \xe1\x88\xb4 abc \xc3\xa5\xc3\xa6\xc3\xa7' # Character 'ሴ' is encoded as 3 bytes.
>>> s1.encode == s1.encode('utf-8') # Encoding 'utf-8' is default.
True
>>> len(s1.encode('utf-8'))
23
>>> len(s1.encode('utf-16')) # 'utf-16' and 'utf-32' are optional encodings.
38
>>> len(s1.encode('utf-32'))
76
from str containing international text
>>> s1 = 'Γ γ Δ δ Ζ ζ Ξ ξ' # Greek
>>> s1.encode
b'\xce\x93 \xce\xb3 \xce\x94 \xce\xb4 \xce\x96 \xce\xb6 \xce\x9e \xce\xbe' # Greek encoded
>>> len(s1)
15 # 8 Greek characters + 7 spaces
>>> len(s1.encode)
23 # 8*2 + 7
Each Greek character occupies 2 bytes and is encoded as 2 bytes. Note for example:
>>> hex(ord('ξ'))
'0x3be'
>>> chr(0x3be)
'ξ'
>>> chr(0x3be).encode
b'\xce\xbe'
>>> s1 = 'А а Б б В в Г г Щ щ Я я' # Cyrillic
>>> s1.encode
b'\xd0\x90 \xd0\xb0 \xd0\x91 \xd0\xb1 \xd0\x92 \xd0\xb2 \xd0\x93 \xd0\xb3 \xd0\xa9 \xd1\x89 \xd0\xaf \xd1\x8f' # Cyrillic encoded.
>>> s1 = 'A Α А' # English 'A', Greek 'Α', Cyrillic 'А'
>>> s1.encode
b'A \xce\x91 \xd0\x90'
>>> s1 = 'ウ ィ キ ペ デ ィ ア' # Japanese
>>> s1.encode
b'\xe3\x82\xa6 \xe3\x82\xa3 \xe3\x82\xad \xe3\x83\x9a \xe3\x83\x87 \xe3\x82\xa3 \xe3\x82\xa2' # Japanese encoded.
>>> len(s1)
13 # 7 Japanese characters + 6 spaces.
>>> len(s1.encode)
27 # 7*3 + 6
>>> s1
'Ξ ξ ウ ィ Щ щ' # Mixture.
>>> s1.encode
b'\xce\x9e \xce\xbe \xe3\x82\xa6 \xe3\x82\xa3 \xd0\xa9 \xd1\x89'
>>> len(s1)
11 # characters
>>> len(s1.encode)
19 # bytes
from str containing hexadecimal digits
classmethod bytes.fromhex(string) returns a bytes object, decoding the given string object:
>>> bytes.fromhex( ' 12 04 E6 d5 ' )
b'\x12\x04\xe6\xd5' # ASCII whitespace in string is ignored provided that hex digits are grouped as bytes.
>>> bytes.fromhex( ' 1204 E6 d5 ' )
b'\x12\x04\xe6\xd5'
>>> bytes.fromhex( ' 12 04E6d5 ' )
b'\x12\x04\xe6\xd5'
>>> bytes.fromhex( ' 41 42 432044 4546 ' ) # To include a space, add '20' meaning b'\x20' or b' '.
b'ABC DEF'
>>> bytes.fromhex( ' 12 20 04 20 E6 20 d5 ' )
b'\x12 \x04 \xe6 \xd5'
>>> bytes.fromhex( ' 12 20 04 20 E6 20 d ' ) # The string must contain exactly two hexadecimal digits per byte.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: non-hexadecimal number found in fromhex arg at position 24
classmethod bytes.fromhex(string) can be used to convert from positive int to bytes object:
>>> i1 = 0xF12B4 ; i1
987828
>>> h1 = hex(i1)[2:] ; h1
'f12b4'
>>> h2 = ('0' * ((len(h1))%2)) + h1 ; h2 # Prepend '0' if necessary.
'0f12b4' # Length is even.
>>> b1 = bytes.fromhex( h2 ) ; b1
b'\x0f\x12\xb4'
Some technical information about encoding standard 'utf-8'
Strings encoded according to encoding standard 'utf-8' conform to the following table:
RangeEncodingDescription
U-00000000 ... U-0000007F0xxxxxxxMarker bit '0' and 7 payload bits (suitable for ASCII)
U-00000080 ... U-000007FF110xxxxx 10xxxxxxMarker bits '110', '10' and 11 payload bits
U-00000800 ... U-0000FFFF1110xxxx 10xxxxxx 10xxxxxxMarker bits '1110', '10', '10' and 16 payload bits
U-00010000 ... U-0010FFFF11110xxx 10xxxxxx 10xxxxxx 10xxxxxxMarker bits '11110', '10', '10', '10' and 21 payload bits
Encoding standard 'utf-8' is a good choice for default encoding because:
2, 3 or 4 bytes are used only if necessary,
it doesn't depend on byte ordering, big or little, and
arbitrary binary data is not likely to conform to the above specification.
Examples of characters encoded with 'utf-8'
>>> bin(ord('Q'))
'0b101_0001' # 7 bits
>>> 'Q'.encode
b'Q' # ASCII fits into 1 byte.
>>> bin(ord('Ξ')) # Greek
'0b11_1001_1110'
>>> ord('Ξ').bit_length
10
>>> 'Ξ'.encode
b'\xce\x9e'
# 1100_1110_1001_1110 0xCE_9E
# 110_01110,10_011110, markers '110' and '10', payload bits 01110_011110 or 011_1001_1110 0x39E
>>> c1 = '先' # Chinese
>>> len(c1)
1
>>> bin(ord(c1))
'0b101_0001_0100_1000'
>>> ord(c1).bit_length
15
>>> c1.encode
b'\xe5\x85\x88'
# 1110_0101_1000_0101_1000_1000 0xE5_85_88
# 1110_0101,10_000101,10_001000, 3 markers and payload bits 0101_000101_001000 or 0101_0001_0100_1000 0x5148
The following code examines chr(0x10006), encoded in 4 bytes:
c1 = chr(0x10006)
print ('c1 = ', c1, '\nord(c1) = ', hex(ord(c1)), sep='')
c1_encoded = c1.encode
print ('c1_encoded =', c1_encoded)
print ([hex(p) for p in c1_encoded], '# each byte of c1_encoded')
print (
The marker bits:
c1_encoded[0] & 0b11111_000 == 0b11110_000 :
c1_encoded[1] & 0b11_000000 == 0b10_000000 :
c1_encoded[2] & 0b11_000000 == 0b10_000000 :
c1_encoded[3] & 0b11_000000 == 0b10_000000 :
'''.format(
c1_encoded[0] & 0b11111_000 == 0b11110_000 ,
c1_encoded[1] & 0b11_000000 == 0b10_000000 ,
c1_encoded[2] & 0b11_000000 == 0b10_000000 ,
c1_encoded[3] & 0b11_000000 == 0b10_000000
# Produce the payload bits:
mask0 = 0b111; mask123 = 0b11_1111
payload = [c1_encoded[0] & mask0]
for p in range (1,4) : payload += [c1_encoded[p] & mask123]
print (
The payload bits:
payload[0] = c1_encoded[0] & 0x07 = & 0x07 =
payload[1] = c1_encoded[1] & 0x3F = & 0x3F =
payload[2] = c1_encoded[2] & 0x3F = & 0x3F =
payload[3] = c1_encoded[3] & 0x3F = & 0x3F =
'''.format(
hex(c1_encoded[0]), hex(payload[0]),
hex(c1_encoded[1]), hex(payload[1]),
hex(c1_encoded[2]), hex(payload[2]),
hex(c1_encoded[3]), hex(payload[3])
s1 = 'payload[3] + (payload[2] << 6) + (payload[1] << 12) + (payload[0] << 18)'
i1 = eval(s1)
print (
Building c1:
i1 = =
i1 == ord(c1) :
'''.format(
s1, hex(i1),
i1 == ord(c1)
c1 = 𐀆
ord(c1) = 0x10006
c1_encoded = b'\xf0\x90\x80\x86'
['0xf0', '0x90', '0x80', '0x86'] # each byte of c1_encoded
The marker bits:
c1_encoded[0] & 0b11111_000 == 0b11110_000 : True
c1_encoded[1] & 0b11_000000 == 0b10_000000 : True
c1_encoded[2] & 0b11_000000 == 0b10_000000 : True
c1_encoded[3] & 0b11_000000 == 0b10_000000 : True
The payload bits:
payload[0] = c1_encoded[0] & 0x07 = 0xf0 & 0x07 = 0x0
payload[1] = c1_encoded[1] & 0x3F = 0x90 & 0x3F = 0x10
payload[2] = c1_encoded[2] & 0x3F = 0x80 & 0x3F = 0x0
payload[3] = c1_encoded[3] & 0x3F = 0x86 & 0x3F = 0x6
Building c1:
i1 = payload[3] + (payload[2] << 6) + (payload[1] << 12) + (payload[0] << 18) = 0x10006
i1 == ord(c1) : True
Theoretically 21 payload bits can contain '\U001FFFFF' but the standard stops at '\U0010FFFF':
>>> chr(0x110006)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: chr arg not in range(0x110000)
A disadvantage of 'utf-8'
A bytes object produced with encoding 'utf-8' can contain the null byte b'\x00'.
This could cause a problem if you are sending a stream of bytes
through a filter that interprets b'\x00' as end of data. Standard 'utf-8'
never produces b'\xFF'. If your bytes object must not contain
b'\x00' after encoding, you could convert the null byte to b'\xFF', then convert
b'\xFF' to b'\x00' before decoding:
>>> s1 = 'Ξ ξ a\000bc \000 Я я 建 页' ; s1
'Ξ ξ a\x00bc \x00 Я я 建 页'
>>> b1 = s1.encode ;b1
b'\xce\x9e \xce\xbe a\x00bc \x00 \xd0\xaf \xd1\x8f \xe5\xbb\xba \xe9\xa1\xb5'
>>> {'Found 0' for p in b1 if p == 0}
{'Found 0'}
# Convert b'\x00' to b'\xFF'
>>> b2 = bytes([ (p,0xFF)[p == 0] for p in b1 ]) ; b2
b'\xce\x9e \xce\xbe a\xffbc \xff \xd0\xaf \xd1\x8f \xe5\xbb\xba \xe9\xa1\xb5'
>>> {'Found 0' for p in b2 if p == 0}
set
>>> # Check conversion from b1 to b2:
>>> {(p == (0,0xFF)) for p in zip(b1,b2) if p[0] != p[1]} # Difference between b1 and b2.
{True}
>>> b2.decode # b2 is not standard 'utf-8'.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 7: invalid start byte
# Before decoding convert b'\xFF' to b'\x00'.
>>> b3 = bytes([ (p,0)[p == 0xFF] for p in b2 ]) ; b3
b'\xce\x9e \xce\xbe a\x00bc \x00 \xd0\xaf \xd1\x8f \xe5\xbb\xba \xe9\xa1\xb5'
>>> s3 = b3.decode ; s3
'Ξ ξ a\x00bc \x00 Я я 建 页'
>>> s3 == s1
True

## Conversion from bytes object

* to int
* The following code illustrates the process for a positive integer:
* def bytes_to_int (input_bytes) :
* isinstance(input_bytes, bytes) or exit (99)
* if (len(input_bytes) == 0) : return 0
* (input_bytes[0] < 0x80) or exit (98)
* shift = i1 = 0
* for p in range(1, len(input_bytes)+1) :
* i1 += (input_bytes[-p] << shift)
* shift += 8
* return i1
* for b1 in [b'', B"\x00\x00\x00", b'''\x13\xd8''', b"""\x00\xf7\x14"""] :
* i1 = bytes_to_int (b1)
* print ('''b1 =, i1 ='''.format(b1, hex(i1)))
* b1 = b'', i1 = 0x0
* b1 = b'\x00\x00\x00', i1 = 0x0
* b1 = b'\x13\xd8', i1 = 0x13d8
* b1 = b'\x00\xf7\x14', i1 = 0xf714
* Class method int.from_bytes(bytes, ....)
* Class method int.from_bytes(bytes, byteorder, *, signed=False) simplifies the conversion from bytes to int:
* >>> hex( int.from_bytes(b'\x13\xf8', 'big', signed=True) )
* '0x13f8'
* >>> hex( int.from_bytes(b'\xd3\xf8', 'big', signed=True) )
* '-0x2c08'
* >>> hex( int.from_bytes(b'\x00\xd3\xf8', 'big', signed=True) )
* '0xd3f8'
* The following code ensures that the integer produced after encoding and decoding is the same as the original int:
* def int_to_bytes (input) : # input is int.
* num_bits = input.bit_length
* num_bytes = (num_bits + 7) // 8
* if ((num_bits % 8) == 0) : num_bytes += 1
* return input.to_bytes(num_bytes, byteorder='big', signed=True)
* def int_from_bytes (input) : # input is bytes.
* return int.from_bytes(input, byteorder='big', signed=True)
* to str
* If the bytes object contains only characters that fit into one byte:
* >>> b2
* b'\t\n\r\x0e 123 abc \xe5\xe6\xe7'
* >>> [chr(p) for p in b2]
* ['\t', '\n', '\r', '\x0e', ' ', '1', '2', '3', ' ', 'a', 'b', 'c', ' ', 'å', 'æ', 'ç']
* >>> s2 = ''.join([chr(p) for p in b2]) ; s2
* '\t\n\r\x0e 123 abc åæç'
* >>> s2a = b2.decode('Latin-1') ; s2a
* '\t\n\r\x0e 123 abc åæç'
* >>> s2 == s2a
* True
* Method bytes.decode creates a string representing the bytes object bytes decoded:
* >>> b1
* b'\t\n\r\x0e 123 \xe1\x88\xb4 abc \xc3\xa5\xc3\xa6\xc3\xa7'
* >>> s1a = b1.decode ; s1a
* '\t\n\r\x0e 123 ሴ abc åæç'
* It is important to use the correct decoding:
* >>> s1b = b1.decode('utf-16') ; s1b
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0xa7 in position 22: truncated data
* to str containing international text
* >>> b1 = b'\xce\x93 \xce\xb3 \xce\x94 \xce\xb4 \xce\x96 \xce\xb6 \xce\x9e \xce\xbe'
* >>> s1 = b1.decode ; s1
* 'Γ γ Δ δ Ζ ζ Ξ ξ' # Greek decoded
* >>> b1 = b'\xd0\x90 \xd0\xb0 \xd0\x91 \xd0\xb1 \xd0\x92 \xd0\xb2 \xd0\x93 \xd0\xb3 \xd0\xa9 \xd1\x89 \xd0\xaf \xd1\x8f'
* >>> s1 = b1.decode ; s1
* 'А а Б б В в Г г Щ щ Я я' # Cyrillic decoded
* >>> b1 = b'\xe3\x82\xa6 \xe3\x82\xa3 \xe3\x82\xad \xe3\x83\x9a \xe3\x83\x87 \xe3\x82\xa3 \xe3\x82\xa2'
* >>> s1 = b1.decode ; s1
* 'ウ ィ キ ペ デ ィ ア' # Japanese decoded.
* >>> len(s1) ; len(b1)
* 13 # Length of s1 in characters.
* 27 # Length of s1 in bytes.
* It is possible to produce different results depending on encoding/decoding:
* >>> s1 = 'Ð± Ð² Ðµ Ð¶ Ð¿ Ð¢ Ð£ Ð¥ Ð© Ð®'
* >>> b1 = s1.encode('Latin-1') ; b1
* b'\xd0\xb1 \xd0\xb2 \xd0\xb5 \xd0\xb6 \xd0\xbf \xd0\xa2 \xd0\xa3 \xd0\xa5 \xd0\xa9 \xd0\xae'
* >>> s1a = b1.decode ; s1a
* 'б в е ж п Т У Х Щ Ю' # Cyrillic.
* >>> s3
* 'ç»´ å¸® å»º é¡µ ä»» è®¡ ä¸ª'
* >>> b3 = s3.encode('Latin-1') ; b3
* b'\xe7\xbb\xb4 \xe5\xb8\xae \xe5\xbb\xba \xe9\xa1\xb5 \xe4\xbb\xbb \xe8\xae\xa1 \xe4\xb8\xaa'
* >>> s3a = b3.decode ; s3a
* '维 帮 建 页 任 计 个' # Chinese
* >>> s3 == s3a
* False # As strings s3 and s3a are not equal. However, they are equal as bytes.
* to str containing hexadecimal digits
* method bytes.hex returns a string object containing two hexadecimal digits for each byte in the instance.
* >>> b'\xf0\xf1\xf2'.hex
* 'f0f1f2'
* >>> b'\xf0\xf1\xf2'*3.hex
* File "<stdin>", line 1
* b'\xf0\xf1\xf2'*3.hex
* SyntaxError: invalid syntax
* >>> (b'\xf0\xf1\xf2'*3).hex # Use parentheses to enforce correct syntax.
* 'f0f1f2f0f1f2f0f1f2'
* >>> (b'\xf0\xf1\xf2'*3)[3].hex # Individual member is returned as int.
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* AttributeError: 'int' object has no attribute 'hex'
* >>> (b'\xf0\xf1\xf2'*3)[3:4].hex # A slice containing 1 byte.
* 'f0'
* method bytes.hex can be used to convert from bytes object to positive int:
* >>> b1 = b'\xF0\x23\x95' ; b1
* b'\xf0#\x95'
* >>> h1 = b1.hex ; h1
* 'f02395'
* >>> i1 = int(h1, 16) ; i1
* 15737749
* >>> hex(i1)
* '0xf02395'

## Operations with methods on bytes objects

* Operations on strings usually require str arguments. Similarly, operations on bytes objects usually require
* bytes arguments. Occasionally, a suitable int may be substituted.
* The following methods on bytes are representative of methods described in the reference. All can be used with arbitrary binary data.
* bytes.count(sub[, start[, end]])
* >>> b'abcd abcd abcd'.count(b' ')
* 2
* >>> b'abcd abcd abcd'.count(b'bcd')
* 3
* >>> b'abcd abcd abcd'[3:].count(b'bcd')
* 2
* >>> b'abcd abcd abcd'[3:10].count(b'bcd')
* 1
* >>> ord('c')
* 99
* >>> b'abcd abcd abcd'.count(99) # chr(99) = 'c'
* 3
* >>> b'\x0a\x0a\x0A'.count(b'\n')
* 3
* >>> b'\x0a\012\n'.count(10)
* 3
* Creating and using a translation table:
* static bytes.maketrans(from, to) returns a translation table to map a byte in from
* into the byte in the same position in to.
* >>> tt1 = bytes.maketrans(b'', bytes(0)) # A translation table without mapping.
* >>> tt2 = bytes([p for p in range(256)]) # Same again.
* >>> tt1 == tt2
* True
* >>> tt1 is tt2
* False
* >>> bytes.maketrans(b'aaaaa', b'1234A') == bytes.maketrans(b'a', b'A') # Duplicates are processed without error.
* True # Last in sequence wins.
* bytes.translate(table, delete=bytes(0)) returns a copy of the bytes object where all bytes occurring in the optional argument delete are removed, and the remaining bytes have been mapped through the given translation table, which must be a bytes object of length 256.
* To invert the case of all alphabetic characters:
* >>> UPPER = b'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
* >>> lower = b'abcdefghijklmnopqrstuvwxyz'
* >>> tt1 = bytes.maketrans(UPPER+lower, lower+UPPER)
* >>> b'The Quick, Brown FOX jumps ....'.translate(tt1)
* b'tHE qUICK, bROWN fox JUMPS ....'
* To delete specified bytes:
* >>> b'The Quick, Brown FOX jumps ....'.translate(None, b'., oO')
* b'TheQuickBrwnFXjumps'
* Deletion is completed before translation:
* >>> b'The Quick, Brown FOX jumps ....'.translate(tt1, b'., o')
* b'tHEqUICKbRWNfoxJUMPS'

## bytes objects and disk files

* Data is written to disk as a stream of bytes. Therefore the bytes object is ideal for this purpose.
* The following code writes a stream of bytes to disk and then reads the data on disk as text.
* Python automatically performs the appropriate decoding (default 'utf-8') when reading text.
* $ cat test.py
* b1 = b'English (cur | prev)'
* b2 = b'Chinese \xef\xbc\x88\xe5\xbd\x93\xe5\x89\x8d | \xe5\x85\x88\xe5\x89\x8d\xef\xbc\x89'
* b3 = b'Japanese (\xe6\x9c\x80\xe6\x96\xb0 | \xe5\x89\x8d)'
* b4 = b'Greek (\xcf\x80\xce\xb1\xcf\x81\xcf\x8c\xce\xbd | \xcf\x80\xcf\x81\xce\xbf\xce\xb7\xce\xb3.)'
* b5 = b'Russian (\xd1\x82\xd0\xb5\xd0\xba\xd1\x83\xd1\x89. | \xd0\xbf\xd1\x80\xd0\xb5\xd0\xb4.)'
* number_written = 0
* try:
* with open("test.bin", "wb") as ofile: # Write bytes to disk.
* for bx in b1,b2,b3,b4,b5 :
* nw = ofile.write(bx + b'\n') # in bytes
* number_written += nw
* end_of_file = ofile.tell # in bytes
* except:
* print ('Error1 detected in "with" statement.')
* (number_written == end_of_file) or exit (99)
* try:
* with open("test.bin", "rt") as ifile: # Read characters from disk.
* for line in ifile :
* print (len(line), line, end='')
* except:
* print ('Error2 detected in "with" statement.')
* exit (0)
* $ python3.6 test.py >test.sout 2>test.serr
* $ od -t x1 test.bin # The contents of disk file test.bin (edited for clarity):
* 0000000 E n g l i s h ' ' ( c u r ' ' | ' ' p # English
* 0000016 r e v )'\n'
* C h i n e s e ' ' ef bc 88 # Chinese
* 0000032 e5 bd 93 e5 89 8d ' ' | 20 e5 85 88 e5 89 8d ef
* 0000048 bc 89'\n'
* J a p a n e s e ' ' ( e6 9c 80 # Japanese
* 0000064 e6 96 b0 ' ' | ' ' e5 89 8d )'\n'
* G r e e k # Greek
* 0000080 ' ' ( cf 80 ce b1 cf 81 cf 8c ce bd ' ' | ' ' cf
* 0000096 80 cf 81 ce bf ce b7 ce b3 . )'\n'
* R u s s # Russian
* 0000112 i a n ' ' ( d1 82 d0 b5 d0 ba d1 83 d1 89 .
* 0000128 ' ' | ' ' d0 bf d1 80 d0 b5 d0 b4 . )'\n'
* 0000142 # Values in left hand column are decimal.
* $ ls -la test.bin
* -rw-r--r-- 1 user staff 142 Nov 12 08:46 test.bin
* $ cat test.bin
* English (cur | prev)
* Chinese （当前 | 先前）
* Japanese (最新 | 前)
* Greek (παρόν | προηγ.)
* Russian (текущ. | пред.)
* $ cat test.sout
* 21 English (cur | prev)
* 18 Chinese （当前 | 先前）# 18 characters including '\n'
* 18 Japanese (最新 | 前)
* 23 Greek (παρόν | προηγ.)
* 25 Russian (текущ. | пред.)

## bytearrays

* The bytearray is a mutable sequence of bytes, similar to the bytes object in that each member
* of the bytearray fits into one byte, and similar to a list in that the bytearray or any slice of it
* may be changed dynamically.
* bytearray displayed
* The bytearray is displayed as a bytes object within parentheses
* prepended by the word bytearray:
* >>> ba1 = bytearray ; ba1 ; len(ba1) ; isinstance(ba1, bytearray)
* bytearray(b'') # An empty bytearray.
* 0
* True
* >>> ba1 = bytearray(3) ; ba1 ; len(ba1) ; isinstance(ba1, bytearray)
* bytearray(b'\x00\x00\x00')
* 3
* True
* >>> ba1 = bytearray([3]) ; ba1 ; len(ba1) ; isinstance(ba1, bytearray)
* bytearray(b'\x03')
* 1
* True
* >>> ba1 = bytearray([3,4,5,6]) ; ba1 ; len(ba1) ; isinstance(ba1, bytearray)
* bytearray(b'\x03\x04\x05\x06')
* 4
* True
* >>> ba1 = bytearray(B'''\x00\001\x02\003''') ; ba1 ; len(ba1) ; isinstance(ba1, bytearray)
* bytearray(b'\x00\x01\x02\x03')
* 4
* True
* >>> ba1 = bytearray(b"""\040abc \x33\x34\x35 \130\131\132\x20""") ; ba1 ; len(ba1) ; isinstance(ba1, bytearray)
* bytearray(b' abc 345 XYZ ')
* 13
* True
* Individual member is returned as int:
* >>> ba1[3] ; chr(ba1[3])
* 99
* 'c'
* Slices of bytearray ba1:
* >>> ba1[1:2]
* bytearray(b'a') # A slice containing 1 byte.
* >>> ba1[4:8]
* bytearray(b' 345') # A slice of 4 bytes.
* >>> ba1[7:3:-1]
* bytearray(b'543 ') # Above slice reversed.
* >>> ba1[::-1]
* bytearray(b' ZYX 543 cba ') # ba1 reversed.
* >>> ba1[4:8] == (ba1[7:3:-1])[::-1]
* True
* bytearray initialized
* Any bytes object may be converted to a bytearray:
* >>> ba1 = bytearray(b'\x54\x68\x65 \161\165\151\143\153, brown \x2E\x2e\056\056')
* >>> ba1; len(ba1) ; isinstance(ba1, bytearray)
* bytearray(b'The quick, brown ....')
* 21
* True
* >>> ba1 = bytearray ( B""" line1
* ... \v \f \n \t line2
* ... line3""")
* >>> ba1
* bytearray(b' line1\n\x0b \x0c \n \t line2\n\t\t\tline3')
* >>> ba2 = bytearray(rB'\x41\x42') ; ba2 ; len(ba2) # Note the rB'..'.
* bytearray(b'\\x41\\x42')
* 8
* >>> isinstance(ba2, bytearray)
* True
* >>> b1 = b'\x5C' + b'x41' + B"""\134""" + b'x42' ; b1
* b'\\x41\\x42'
* >>> isinstance(b1, bytes)
* True
* >>> b1 == ba2 # Comparing bytes object and bytearray.
* True
* A suitable sequence can be converted to bytearray:
* >>> ba1 = bytearray([16, 0x20, ord('a'), 98, 0x63, 0o144, ord(' '), 0xEe]) # Input to bytearray is list.
* >>> ba1; len(ba1) ; isinstance(ba1, bytearray)
* bytearray(b'\x10 abcd \xee')
* 8
* True
* >>> bytearray( [ ord(p) for p in 'xyz £ § « ® ½ Ø ñ' ] )
* bytearray(b'xyz \xa3 \xa7 \xab \xae \xbd \xd8 \xf1')
* Concatenation of bytearray and bytes object:
* >>> bytearray(b'abc') + B''' ''' + b"""\x44\x45\x46""" + b"\040" + B'\06123'
* bytearray(b'abc DEF 123')
* Because the bytearray is a mutable sequence, the bytearray accepts assignment:
* >>> ba1 = bytearray(b'The quick, brown ') ; ba1
* bytearray(b'The quick, brown ')
* >>> ba1 += b'fox' ; ba1
* bytearray(b'The quick, brown fox')
* >>> ba1[4:9] = b'lazy' ; ba1
* bytearray(b'The lazy, brown fox')
* >>> ba1[4:8] = ba1[4:8].upper ; ba1
* bytearray(b'The LAZY, brown fox')
* >>> ba1[4:10] = b'' ; ba1
* bytearray(b'The brown fox')
* bytearray as iterable
* >>> ba1 = bytearray(b'The quick, brown ')
* >>> if b'quick' in ba1 : print ('found')
* found
* >>> if bytearray(b'quick') in ba1 : print ('found')
* found
* >>> if 32 in ba1 : print ('found') # chr(32) = ' '.
* found
* >>> for p in ba1 :
* ... print (p,' ',end='')
* 84 104 101 32 113 117 105 99 107 44 32 98 114 111 119 110 32 >>>
* >>> for p in ba1.upper :
* ... print (chr(p),end='')
* THE QUICK, BROWN >>>

## Conversion to bytearray

* from int
* The following code illustrates the process for a positive integer:
* >>> I1 = 0x18b9e4
* >>> bytearray ( [ (I1 >> 16) & 255, (I1 >> 8) & 255, I1 & 255 ] )
* bytearray(b'\x18\xb9\xe4')
* Method int.to_bytes(length, ....)
* Method int.to_bytes(length, byteorder, *, signed=False) returns a bytes object representing an integer.
* If a bytearray is required, convert the bytes object to bytearray.
* >>> bytearray ( 0x12_84.to_bytes(2, 'big', signed=True) )
* bytearray(b'\x12\x84')
* >>> bytearray ( (-0xE2_04).to_bytes(3, 'big', signed=True) ) # Note the parentheses: (-0xE2_04).
* bytearray(b'\xff\x1d\xfc') # 3 bytes for a signed, negative int.
* from str
* If each character of the string fits into one byte, the process is simple:
* >>> s1 = '\011\012\015\016 123 abc \345\346\347' ; s1
* '\t\n\r\x0e 123 abc åæç'
* >>> ba1 = bytearray( [ ord(p) for p in s1 ] ) ; ba1
* bytearray(b'\t\n\r\x0e 123 abc \xe5\xe6\xe7')
* >>> s1.encode('Latin-1') == ba1 # Comparing bytes object and bytearray.
* True
* Method str.encode creates a bytes object containing the string str encoded. If a
* bytearray is required, convert the bytes object to bytearray.
* >>> s1 = '\t\n\r\x0e 123 ሴ abc åæç' ; s1
* '\t\n\r\x0e 123 ሴ abc åæç'
* >>> bytearray ( s1.encode ) # Each character 'åæç' occupies one byte but is encoded as 2 bytes.
* bytearray(b'\t\n\r\x0e 123 \xe1\x88\xb4 abc \xc3\xa5\xc3\xa6\xc3\xa7') # Character 'ሴ' is encoded as 3 bytes.
* >>> bytearray ( s1.encode ) == s1.encode('utf-8')
* True
* from str containing international text
* >>> s1
* 'Ξ ξ ウ ィ Щ щ' # Mixture of Greek, Japanese, Russian.
* >>> ba1 = bytearray ( s1.encode ) ; ba1
* bytearray(b'\xce\x9e \xce\xbe \xe3\x82\xa6 \xe3\x82\xa3 \xd0\xa9 \xd1\x89')
* >>> len(s1)
* 11 # characters
* >>> len(ba1)
* 19 # bytes
* >>> ba1 == s1.encode('utf-8')
* True
* from str containing hexadecimal digits
* classmethod bytearray.fromhex(string) returns a bytearray, decoding the given string object:
* >>> bytearray.fromhex( ' 12 04 E6 d5 ' )
* bytearray(b'\x12\x04\xe6\xd5') # ASCII whitespace in string is ignored provided that hex digits are grouped as bytes.
* >>> bytearray.fromhex( ' 12 20 04 20 E6 20 d5 ' )
* bytearray(b'\x12 \x04 \xe6 \xd5')
* >>> bytearray.fromhex( ' 12 20 04 20 E6 20 d ' ) # The string must contain exactly two hexadecimal digits per byte.
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* ValueError: non-hexadecimal number found in fromhex arg at position 24
* classmethod bytearray.fromhex(string) can be used to convert from positive int to bytearray:
* >>> i1 = 0xF12B4 ; i1
* 987828
* >>> h1 = hex(i1)[2:] ; h1
* 'f12b4'
* >>> h2 = ('0' * ((len(h1)) & 1)) + h1 ; h2 # Prepend '0' if necessary.
* '0f12b4' # Length is even.
* >>> ba1 = bytearray.fromhex( h2 ) ; ba1
* bytearray(b'\x0f\x12\xb4')

## Conversion from bytearray

* to int
* The following code illustrates the process for a positive integer:
* >>> ba1
* bytearray(b'\x0f\x12\xb4')
* >>> i1 = (ba1[0] << 16) + (ba1[1] << 8) + ba1[2] ; hex(i1)
* '0xf12b4'
* Class method int.from_bytes(bytes, ....)
* Class method int.from_bytes(bytes, byteorder, *, signed=False) simplifies the conversion from bytearray to int:
* >>> hex( int.from_bytes(bytearray(b'\xd3\xf8'), 'big', signed=True) )
* '-0x2c08'
* >>> hex( int.from_bytes(bytearray(b'\x00\xd3\xf8'), 'big', signed=True) )
* '0xd3f8'
* to str
* If the bytearray contains only characters that fit into one byte:
* >>> ba2 = bytearray(b'\t\n\r\x0e 123 abc \xe5\xe6\xe7') ; ba2
* bytearray(b'\t\n\r\x0e 123 abc \xe5\xe6\xe7')
* >>> s2 = ''.join([chr(p) for p in ba2]) ; s2
* '\t\n\r\x0e 123 abc åæç'
* >>> s2a = ba2.decode('Latin-1') ; s2a
* '\t\n\r\x0e 123 abc åæç'
* >>> s2 == s2a
* True
* Method bytearray.decode creates a string representing the bytearray decoded:
* >>> ba1 = bytearray(b'\t\n\r\x0e 123 \xe1\x88\xb4 abc \xc3\xa5\xc3\xa6\xc3\xa7') ; ba1
* bytearray(b'\t\n\r\x0e 123 \xe1\x88\xb4 abc \xc3\xa5\xc3\xa6\xc3\xa7')
* >>> s1 = ba1.decode ; s1
* '\t\n\r\x0e 123 ሴ abc åæç'
* It is important to use the correct decoding:
* >>> s1b = ba1.decode('utf-16') ; s1b
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0xa7 in position 22: truncated data
* to str containing international text
* >>> ba1 = bytearray(b'\xce\x93 \xce\xb3 \xce\x94 \xce\xb4 \xce\x96 \xce\xb6')
* >>> s1 = ba1.decode ; s1
* 'Γ γ Δ δ Ζ ζ' # Greek decoded.
* >>> ba2 = bytearray(b'\xd0\x90 \xd0\xb0 \xd0\x91 \xd0\xb1 \xd0\x92 \xd0\xb2 \xd0\x93 \xd0\xb3 \xd0\xa9 \xd1\x89')
* >>> s2 = ba2.decode ; s2
* 'А а Б б В в Г г Щ щ' # Cyrillic decoded.
* >>> ba3 = bytearray(b'\xe3\x82\xa6 \xe3\x82\xa3 \xe3\x82\xad \xe3\x83\x9a \xe3\x83\x87 \xe3\x82\xa3')
* >>> s3 = ba3.decode ; s3
* 'ウ ィ キ ペ デ ィ' # Japanese decoded.
* >>> len(s3)
* 11 # Length of s3 in characters.
* >>> len(ba3)
* 23 # Length of s3 in bytes.
* to str containing hexadecimal digits
* method bytearray.hex returns a string object containing two hexadecimal digits for each byte in the instance.
* >>> bytearray(b'\xf0\xf1\xf2').hex
* 'f0f1f2'
* >>> (bytearray(b'\xf0\xf1\xf2') * 2).hex
* 'f0f1f2f0f1f2'
* >>> (bytearray(b'\xf0\xf1\xf2') * 2)[1:4].hex
* 'f1f2f0'
* method bytearray.hex can be used to convert from bytearray to positive int:
* >>> ba1 = bytearray(b'\xF0\x23\x95') ; ba1
* bytearray(b'\xf0#\x95')
* >>> h1 = ba1.hex ; h1
* 'f02395'
* >>> i1 = int(h1, 16) ; i1 ; hex(i1)
* 15737749
* '0xf02395'

## Operations with methods on bytearrays

* The following methods on bytearrays are representative of methods described in the reference.
* All can be used with bytes objects.
* bytearray.find(sub[, start[, end]])
* >>> bytearray([5,6,7,8,9]).find(8)
* 3
* >>> bytearray([5,6,7,8,9]).find(b'\x07\x08')
* 2
* >>> bytearray([5,6,7,8,9]).find(23)
* -1
* bytearray.join(iterable)
* >>> bytearray(b' ').join([b'123', bytes([55,56,57]), bytearray(b'XYZ')])
* bytearray(b'123 789 XYZ')
* bytearray.partition(sep)
* >>> bytearray([8,9,10,11,12]).partition(b'\x0a')
* (bytearray(b'\x08\t'), bytearray(b'\n'), bytearray(b'\x0b\x0c'))
* >>> bytearray(b'abcdefg').partition(b'c')
* (bytearray(b'ab'), bytearray(b'c'), bytearray(b'defg'))
* >>> bytearray(b'abcdefg').partition(b'X')
* (bytearray(b'abcdefg'), bytearray(b''), bytearray(b''))
* >>> bytearray(b'abcdefg').partition(b'cde')
* (bytearray(b'ab'), bytearray(b'cde'), bytearray(b'fg'))
* bytearray.center(width[, fillbyte])
* >>> bytearray(b'abcd').center(10)
* bytearray(b' abcd ')
* >>> bytearray(b'abcd').center(10, b'.')
* bytearray(b'...abcd...')
* bytearray.split(sep=None, maxsplit=-1)
* >>> bytearray(b'1 2 3 4 5').split
* [bytearray(b'1'), bytearray(b'2'), bytearray(b'3'), bytearray(b'4'), bytearray(b'5')]
* >>> bytearray(b'1 2 3 4 5').split(b' ')
* [bytearray(b'1'), bytearray(b'2'), bytearray(b''), bytearray(b'3'), bytearray(b''), bytearray(b''), bytearray(b'4'), bytearray(b''), bytearray(b''), bytearray(b''), bytearray(b''), bytearray(b'5')]
* >>> bytearray(b'1 2 3 4 5').split(b'4')
* [bytearray(b'1 2 3 '), bytearray(b' 5')]
* bytearray.strip([chars])
* >>> bytearray(b' \v \f \n \t ').strip
* bytearray(b'')
* >>> bytearray(b' 123 ').strip
* bytearray(b'123')
* >>> bytearray(b'www.example.com').strip(b'cmwz.')
* bytearray(b'example.co')
* bytearray.isspace
* >>> bytearray(b'').isspace
* False
* >>> bytearray(b' \v \f \n \t ').isspace
* True
* >>> bytearray(b' \v \f \n \t x ').isspace
* False

## Further Reading or Review

* Previous Lesson: Lists
* This Lesson: Bytes objects and Bytearrays
* Next Lesson: Tuples
* Course Home Page