Working with bytes in Dart
Tags: Dart, Flutter, Programming, Binary

If you can understand the bytes, you can understand anything.

Preface

I started on this topic because I was researching how to communicate between Dart and a PostgreSQL database server. It turned out to be much more low-level than I had been expecting. I thought I’d write a short article explaining some of new things I was learning. Well, three full days of writing later, that short article has turned into one of the most in-depth explanations I’ve ever written on an already niche topic. Although the article is long, I don’t think you’ll find it boring, and there’s a good chance you learn one or two new things about Dart even if you’ve been using it for a while. I certainly did. As always, please let me know if you find any errors. That’s how I learn. And try the code examples out yourself. That’s how you learn.

This article is up to date for Dart 2.10.

Bits and bytes

Everyone reading this article knows that a byte is eight bits:

00101010

That 8-bit byte has a value, a value of 42 in this case, which is just an integer. Now combining that knowledge with the fact that all binary data is just a sequence of bytes, this means it’s possible to represent any binary data as a list of integers in Dart:

List<int> data = [102, 111, 114, 116, 121, 45, 116, 119, 111, 0];

Those bytes might be from a file, a bitmap image, an mp3 recording, a memory dump, a network request, or the character codes of a string. They’re all bytes.

Being a little more efficient

In Dart the int type defaults to a 64-bit value. That’s eight bytes. Here’s the number 42 again, this time showing all 64 bits for your visual reference:

0000000000000000000000000000000000000000000000000000000000101010

If you look carefully, you might notice that a lot of those bits aren’t being used.

An int can store values as large as 9,223,372,036,854,775,807, but the largest a byte will ever be is 255. This is definitely a case of using a sledgehammer to crack a nut. Now imagine a modest one-megabyte file as a list of int s. Your problem just got a million times bigger.

This is where Uint8List comes in. This type is basically like List<int> , but Uint8List is much more efficient than List<int> for large lists. Uint8List is a list of integers where the values in the list are only 8 bits each, or one byte. The U of Uint8List means unsigned, so the values range from 0 to 255 . That’s perfect for representing binary data!

Side note: Negative numbers in binary

Have you ever though about how to represent negative numbers in binary? Well, the way to do it is to make the left most bit a 1 for negative numbers and a 0 for positive numbers. For example, these 8-bit signed integers are all negative because they all start with 1 :

11111101

10000001

10101010

On the other hand, the following 8-bit signed integers are all positive because the left most bit is 0 :

01111111

00000001

01010101

You might think that if 00000001 is +1 , then 10000001 should be -1 . It doesn’t quite work like that, though. Otherwise you’d have two values for zero: 10000000 and 00000000 . The solution is to use a system called Two’s Complement. The following video explains it very nicely if your interested.

In Dart you can already use the int type to represent signed values, both positive and negative. However, if you want lists of only 8-bit signed integers, you can use the Int8List type (note the absence of U ). This allows values from -128 to 127 . That’s not particularly useful for most use cases of represent byte data, though, so we’ll stick with the unsigned integers in Uint8List .

Converting List<int> to Uint8List

Uint8List is part of dart:typed_data , one of the core Dart libraries. To use it, add the following import:

import 'dart:typed_data';

You can now convert the List<int> list you had earlier to Uint8List by using the fromList method:

List<int> data = [102, 111, 114, 116, 121, 45, 116, 119, 111, 0];

Uint8List bytes = Uint8List.fromList(data);

If any of the values in the List<int> list are outside of the 0 to 255 range, then the values will wrap. You can see that in the following example:

List<int> data = [3, 256, -2, 2348738473];

Uint8List bytes = Uint8List.fromList(data);

print(bytes); // [3, 0, 254, 169]

256 went one beyond 255 so it became 0 . The next number, -2 , is less than 0 so it wrapped around to the top by two positions and became 254 . And I have no idea how many times 2348738473 wrapped, but eventually it ended up at 169 .

If you don’t want this wrapping to occur, you can use Uint8ClampedList instead. This will clamp all values larger than 255 to 255 and all values less than 0 to 0 .

List<int> data = [3, 256, -2, 2348738473];

Uint8ClampedList bytes = Uint8ClampedList.fromList(data);

print(bytes); // [3, 255, 0, 255]

This time the final three values were all clamped.

Creating a Uint8List

In the method above you created a Uint8List by converting a List<int> . What if you just want to start with a byte list, though? You can do that by passing in the length of the list into the constructor like so:

final byteList = Uint8List(128);

This creates a fixed length list where all 128 values are 0 . Print byteList and you’ll see:

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Feel free to count them to make sure there really are 128.

Modifying a byte list

How do you modify the values of a Uint8List ? Just like you would for a normal List :

byteList[2] = 255;

Printing byteList again shows that the value at index 2 has changed:

[0, 0, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

The random access nature of lists makes it fast and convenient to modify the byte value at any index location.

Growable lists

The list you made above was fixed-length. If you tried to do something like this:

final byteList = Uint8List(128);

byteList.add(42);

you’d get the following exception:

Unsupported operation: Cannot add to a fixed-length list

That’s not very convenient if you’re trying to do something like collect the bytes from a stream. To make a growable list of bytes, you need a BytesBuilder :

final bytesBuilder = BytesBuilder();

bytesBuilder.addByte(42);

bytesBuilder.add([0, 5, 255]);

You can add single bytes (of type int ) or lists of bytes (of type List<int> ).

When you want to convert the builder to Uint8List , use the toBytes method like so:

Uint8List byteList = bytesBuilder.toBytes();

print(byteList); // [42, 0, 5, 255]

More than one way to look at bits

Bits and bytes are just a lot of zeros and ones. They don’t have much meaning if you don’t have a way of interpreting them:

0011111100011101110011011101101110011111100110011011110000010101001110100010101010000111100101101111001111011100000101011001101111010101101000011101100101000111100000100100011111101011111110100111010011001100000000000111111010010010000000001111010001100000

With Uint8List the way we interpret them is to say that every 8 bits is a number between 0 and 255:

00111111 00011101 11001101 11011011 10011111 10011001 10111100 00010101 00111010 00101010 10000111 10010110 11110011 11011100 00010101 10011011 11010101 10100001 11011001 01000111 10000010 01000111 11101011 11111010 01110100 11001100 00000000 01111110 10010010 00000000 11110100 01100000

You could reinterpret those same values as signed integers from -128 to 127. That’s what we did with Int8List .

There are also other ways to look at the data, though. For example, rather than using 8-bit chunks, you could look at the data as a list of 16-bit chunks:

0011111100011101 1100110111011011 1001111110011001 1011110000010101 0011101000101010 1000011110010110 1111001111011100 0001010110011011 1101010110100001 1101100101000111 1000001001000111 1110101111111010 0111010011001100 0000000001111110 1001001000000000 1111010001100000

The dart:typed_data library has types for that, too. Use Uint16List for unsigned integers ranging from 0 to 65,535, or Int16List for signed integers ranging from -32,768 to 32,767.

It doesn’t stop there. You could also interpret that same data as a list of 32-bit values:

00111111000111011100110111011011 10011111100110011011110000010101 00111010001010101000011110010110 11110011110111000001010110011011 11010101101000011101100101000111 10000010010001111110101111111010 01110100110011000000000001111110 10010010000000001111010001100000

Or 64-bit values:

0011111100011101110011011101101110011111100110011011110000010101 0011101000101010100001111001011011110011110111000001010110011011 1101010110100001110110010100011110000010010001111110101111111010 0111010011001100000000000111111010010010000000001111010001100000

Or even 128-bit values:

00111111000111011100110111011011100111111001100110111100000101010011101000101010100001111001011011110011110111000001010110011011 11010101101000011101100101000111100000100100011111101011111110100111010011001100000000000111111010010010000000001111010001100000

Dart has types for all of those:

Int32List

Uint32List

Int64List

Uint64List

Int32x4List (128 bits)

Using the specific list type that you need will be more efficient for large amounts of data than using List<int> .

Byte views in Dart

Dart backs the raw binary data with a ByteBuffer . The types that you saw in the last section all implement a class called TypedData , which is just a generic way of viewing the data in the ByteBuffer . That means types like Uint8List , Int32List , and Uint64List are all just different ways of looking at the same data.

We’ll use the following list of four bytes for the examples to come:

00000001 00000000 00000000 10000000

In decimal form the list would look like this:

1, 0, 0, 128

First create the list in Dart as you have done previously:

Uint8List bytes = Uint8List.fromList([1, 0, 0, 128]);

As with any form of TypedData , you can get access to the underlying ByteBuffer of a Uint8List by accessing the buffer property:

ByteBuffer byteBuffer = bytes.buffer;

Unsigned 16-bit view

Now that you have the byte buffer, you can get a different view of the bytes by using one of the as… methods, this time asUint16List :

Uint16List sixteenBitList = byteBuffer.asUint16List();

Before you print sixteenBitList to see the contents, what do you think the values will be?

Think you know the answer? Ok, print the list:

print(sixteenBitList);

On my Mac computer the results are as follows:

[1, 32768]

That’s very strange. Since the original values were:

00000001 00000000 00000000 10000000

1 0 0 128

I would have expected them to be combined into 16-bit chunks like this:

0000000100000000 0000000010000000

256 128

Instead, we got this:

0000000000000001 1000000000000000

1 32768

Hold that thought. Let’s check the 32-bit view.

Unsigned 32-bit view

I’ll start with the a byte list containing the decimal values 0, 1, 2, 3. That way we can see if they are in the same order or not. For clarity, here is what the original list will look like in 8-bit binary form:

00000000 00000001 00000010 00000011

0 1 2 3

Now run the following code:

Uint8List bytes = Uint8List.fromList([0, 1, 2, 3]);

ByteBuffer byteBuffer = bytes.buffer;

Uint32List thirtytwoBitList = byteBuffer.asUint32List();

print(thirtytwoBitList);

This time you have a 32-bit view in the underlying buffer from the original Uint8List . The print statement reveals a value of 50462976 , which in 32-bit binary is:

00000011000000100000000100000000

or if you add spacing (to help see the parts):

00000011 00000010 00000001 00000000

That’s exactly the reverse order of the original! What’s happening?

Endianness

You don’t normally have to think about this kind of thing when all you’re doing is happily building a Flutter app, but when you go as low as we are today, you’re actually bumping up against the machine architecture.

Some machines order the individual bytes within a chunk of bytes (whether it’s 2-byte chunks, 4-byte chunks, or 8-byte chunks) in forward order. This is called big endian because the most significant byte is first. That’s normally what we expect because we read numbers from left to right, starting with the biggest part of the number.

However, other machines order the individual bytes within a chunk of bytes in reverse order. This is called little endian. As non-intuitive as that may seem, it actually works on the machine architecture level. The problem comes when big endian meets little endian. And that’s what happened to us above.

Before I talk more about that, you may like to watch this video on endianness. The explanation is pretty good (if you can get past the little story at the beginning):

Where the miscommunication happened

When I ran the code:

Uint8List bytes = Uint8List.fromList([0, 1, 2, 3]);

ByteBuffer byteBuffer = bytes.buffer;

Uint32List thirtytwoBitList = byteBuffer.asUint32List();

print(thirtytwoBitList);

Dart took the list of these four bytes:

00000000 00000001 00000010 00000011

And gave them to my computer. Dart knew that 00000000 was first, 00000001 was second, 00000010 was third, and 00000011 was last. My computer also knew that.

Then Dart asked my computer for a 32-bit chunk view of that list of bytes from the byte buffer in my computer’s memory. My computer happily oblidged and returned:

00000011000000100000000100000000

Well, it turns out that my MacBook has a little endian architecture. My computer still knew that 00000000 was first, 00000001 was second, 00000010 was third, and 00000011 was last. However, when the print statement called on some toString method somewhere, it interpreted those 32 bits as a single integer in big endian format. Can’t say I blame it.

Checking endianness in Dart

If you want to find out what the endian architecture of your system is, you can run the following code in Dart:

print(Endian.host == Endian.little);

If that prints true , then you too have a little endian machine. Otherwise, it’s big endian. Chances are that yours is little endian, though, since most personal computers nowadays use it.

If you look at the source code for the Endian class, you’ll see that the way it checks the architecture of the host is very similar to what led to my initial surprise:

class Endian {

...

static final Endian host =

(new ByteData.view(new Uint16List.fromList([1]).buffer))

.getInt8(0) == 1

? little

: big;

}

It takes a 16-bit representation of 1 and then sees if the first 8 bits is 1 , in which case it’s little endian. That’s because little endian order those 16 bits like this:

00000001 00000000

while big endian machines order them like this:

00000000 00000001

What do I care?

You might think you don’t need to worry about this since you don’t plan on getting so close to the machine hardware. However, endianness comes up in other contexts, too. Basically anytime you are working with shared data that comes in anything bigger than 8-bit chunks, you need to be aware of endianness.

For example, if you watched the video I linked to above, you know that different file formats use different endian encoding:

JPEG (big endian)

GIF (little endian)

PNG (big endian)

BMP (little endian)

MPEG-4 (big endian)

Network data (big endian)

UTF-16 text files (big or little endian — watch this video)

So even if you know the endianness of the raw byte data you’re working with, how do you communicate that to Dart so that you don’t get the same miscommunication problem that you saw I had earlier?

Keep reading to find out.

Handling endianness in Dart

Once you’ve got your raw data as some TypedData like Uint8List , you can use the ByteData class to do random access read and write tasks on it. This way you don’t have to interact with the BytesBuffer directly. ByteData also allows you to specify the endianness with which you want to interpret the data.

Write the following lines of code:

Uint8List byteList = Uint8List.fromList([0, 1, 2, 3]);

ByteData byteData = ByteData.sublistView(byteList);

int value = byteData.getUint16(0, Endian.big);

print(value);

Here’s what that does:

You get your Uint8List typed data from a list of integers. This is the same as you’ve seen previously.

typed data from a list of integers. This is the same as you’ve seen previously. Then you wrap this with ByteData . A view (or “sublist” view) is a way of looking at the data in the buffer. The reason it’s called a sublist is that you don’t necessary need to have a view of the entire buffer, which could be very large. It’s possible to view a smaller range of bytes within the buffer. In our case, though, the buffer for byteList only includes those four bytes, so we’re viewing the whole thing.

. A view (or “sublist” view) is a way of looking at the data in the buffer. The reason it’s called a sublist is that you don’t necessary need to have a view of the entire buffer, which could be very large. It’s possible to view a smaller range of bytes within the buffer. In our case, though, the buffer for only includes those four bytes, so we’re viewing the whole thing. After that you access 16 bits (or two bytes) starting at index 0 , where the indexing is in one-byte increments. If you chose 1 as the index you’d get the second and third bytes, and if you chose 2 as the index you’d get the third and fourth bytes. Any index higher that 2 in this case would throw an error since there are only four bites available.

, where the indexing is in one-byte increments. If you chose as the index you’d get the second and third bytes, and if you chose as the index you’d get the third and fourth bytes. Any index higher that in this case would throw an error since there are only four bites available. Finally, you also tell Dart that you want to interpret those two bytes starting at index 0 as big endian. That’s actually the default for ByteData , so you could have left that parameter out.

Run the code and you’ll see an output of 1 . That makes sence because

00000000 00000001

interpreted in a big endian way has a value of 1.

Do it again replace Endian.big with Endian.little . Now when you run the code you’ll see 256 because it’s interpreting the second byte ( 00000001 ) as coming before the first one ( 00000000 ) so you get

0000000100000000

which isn’t what you want for this data, so change the code back to Endian.big .

Setting bytes in ByteData

In the section above you were getting bytes. You can also set bytes while specifying the endianness in nearly the same way you got them. Add the following line to what you wrote above:

byteData.setInt16(2, 256, Endian.big);

print(byteList);

Here you are setting two bytes (with a value of 256) beginning at index two in the buffer. Print that and you’ll see:

[0, 1, 1, 0]

Does that make sense? Previously the buffer contained:

[0, 1, 2, 4]

But you replaced the last two bytes with with 256, which in binary is:

00000001 00000000

or 1 and 0 , when interpreted in a big endian way.

OK, that’s enough about endianness. Most of the time you can just accept the default of big endian and not think about it.

Hexadecimal and binary

Up to this point I’ve been writing all the binary numbers in this article as strings like 10101010 , but it’s not very convenient to use that format when writing code because Dart defaults to base 10 for integers.

int x = 10000000;

That’s ten million, not 128. :(

While it may be hard to directly write binary, it’s quite easy to directly write hexadecimal values in Dart. Just prefix the hex number with 0x like so:

int x = 0x80; // 10000000 binary

print(x); // 128 decimal

Since the relationship between binary and hex is so direct, that makes converting between the two an easy task. If you’re working with binary a lot, the conversion table might be worth memorizing:

hex binary

----------

0 0000

1 0001

2 0010

3 0011

4 0100

5 0101

6 0110

7 0111

8 1000

9 1001

A 1010

B 1011

C 1100

D 1101

E 1110

F 1111

Byte values will always be in two hex-value chunks (or nibbles as a 4-bit chunk is technically called, oh the humor of those early programmers).

Here are a few more examples. If you cover the binary and only look at the hex, can you figure out the binary without looking?

hex binary

----------------

80 1000 0000

2A 0010 1010

F2F2F2 1111 0010 1111 0010 1111 0010

1F60E 0001 1111 0110 0000 1110

As a bit of trivia, the third value is the RGB (red green blue) hex value for the shade of grey that Medium uses as the background color for code blocks. And the last one is the Unicode code point of the smiling face with sunglasses emoji 😎.

Converting hex and binary strings

A fun trick in Dart that you may not know about is that you can actually convert between different bases and get the string representation of the values.

Here is an example of taking a decimal number and converting it to hex and binary in string form:

String hex = 2020.toRadixString(16).padLeft(4, '0');

print(hex); // 07e4 String binary = 2020.toRadixString(2);

print(binary); // 11111100100

Notes:

Radix just means the base.

The first example takes the decimal number 2020 , converts it to base-16 (i.e., hexadecimal), and makes sure the length is 4. Any length less than 4 is padded with zeros on the left. This made 7e4 become 07e4 .

, converts it to base-16 (i.e., hexadecimal), and makes sure the length is 4. Any length less than 4 is padded with zeros on the left. This made become . The second example converts the decimal 2020 to binary in String format.

You can do the conversion the other way using int.parse :

int myInt = int.parse('07e4', radix: 16);

print(myInt); // 2020 myInt = int.parse('11111100100', radix: 2);

print(myInt); // 2020

Converting Unicode strings to bytes and back

When I say bytes here, I just mean numbers. The String type in Dart is a list of Unicode numbers. Unicode numbers are called code points and can be as small as 0 or as large as 10FFFF . Here is an example:

character Unicode code point (hex)

-------------------------------

H 48

e 65

l 6C

l 6C

o 6F

😎 1F60E

The word Dart uses for code points is runes. You can see them like this:

Runes codePoints = 'Hello😎'.runes;

print(codePoints); // (72, 101, 108, 108, 111, 128526)

That’s the decimal version of their hex values.

Runes are an iterable collection of int values. However, just as you saw at the beginning of the article with List<int> and Uint8List , 64 or even 32 bits isn’t very efficient to store text when most characters in the English speaking world (besides emoji like 😎) only take 8 bits. Even the majority of the thousands and thousands of Chinese characters that exist can be encoded with less than 16 bits.

For that reason, most people encode Unicode values using an 8-bit or 16-bit encoding system. When a Unicode value is too big to fit in 8 or 16 bits, the system uses a special trick to encode larger values. The 8-bit encoding system is called UTF-8 and the 16-bit system is called UTF-16.

The way UTF-8 and UTF-16 are encoded is super interesting, especially the tricks they use to encode large Unicode code points. If you’re interested in that kind of thing (and you probably are if you’re still reading), you should definitely watch the following video. It’s the best one I’ve ever seen on the topic:

UTF-16 conversion in Dart

Although Unicode code points are available as an Iterable when you ask for runes, Dart uses UTF-16 as its actual encoding for strings internally. These 16-bit values are called code units rather than code points.

The conversion to get the UTF-16 code units from a string is easy:

List<int> codeUnits = 'Hello😎'.codeUnits;

print(codeUnits); // [72, 101, 108, 108, 111, 55357, 56846]

Notes:

You might be thinking, “Hey, an int is 64 bytes, not 16!” That’s just how it’s represented to you externally after you do the conversion. The int type is supposed to be generic and you’re not really supposed to think about how many bytes it uses. Internally, strings are lists of 16-bit integers.

is 64 bytes, not 16!” That’s just how it’s represented to you externally after you do the conversion. The type is supposed to be generic and you’re not really supposed to think about how many bytes it uses. Internally, strings are lists of 16-bit integers. You can see that it took two UTF-16 values to represent 😎: 55357 and 56846 , which is D83D and DE0E in hex. These two numbers are called a surrogate pair, which you’ll know all about if you watched the video above.

Decimal Hex Binary

--------------------------------

55357 D83D 1101100000111101 (high surrogate)

56846 DE0E 1101111000001110 (low surrogate) String Hex Binary

----------------------------------------

F60E 0000 1111 0110 0000 1110

+ 10000 0001 0000 0000 0000 0000

----------------------------------------

😎 1F60E 0001 1111 0110 0000 1110

Since every UTF-16 code unit is the same length, that also makes it super easy to access any code unit by its index:

int index = 0;

int codeUnit = 'Hello😎'.codeUnitAt(index);

print(codeUnit); // 72

The conversion of going from code units back to a String is easy too:

List<int> codeUnits = 'Hello😎'.codeUnits;

final myString = String.fromCharCodes(codeUnits);

print(myString); // Hello😎 print(String.fromCharCode(72)); // H

As nice as all that is, there are some problems with UTF-16 based string manipulation. For example, if you are programmatically deleting one code unit at a time, there is a good chance that you’ll forget about surrogate pairs (and grapheme clusters). Smiley isn’t so smiley anymore when that happens. Read the following articles for more on this topic:

Converting UTF-8 strings

While Dart uses UTF-16 encoding, the internet uses UTF-8 encoding. That means when you are transfering text over the network, you’ll need to convert your Dart strings to and from a list of UTF-8 encoded integers. Again, remember that this is just a list of 8-bit binary numbers, specifically a Uint8List . The bonus with 8-bit values is that you don’t need to worry about endianness.

If you watched the video I linked to above and now understand how UTF-8 encoding works, you could write the encoder and decoder yourself. However, you don’t need to, because Dart already has them in its dart:convert library.

Import the library like this:

import 'dart:convert';

Then you can do the conversion from a string to UTF-8 simply like this:

Uint8List encoded = utf8.encode('Hello😎');

print(encoded); // [72, 101, 108, 108, 111, 240, 159, 152, 142]

This time Smiley is encoded as four 8-bit values:

Decimal Hex Binary

------------------------

240 F0 11110000

159 9F 10011111

152 98 10011000

142 8E 10001110 String Hex Binary

------------------------

😎 1F60E 00001 1111 0110 0000 1110

I’m telling you, you need to watch that video. 😎

Converting from UTF-8 back to a String is just as easy. This time use the decode method:

List<int> list = [72, 101, 108, 108, 111, 240, 159, 152, 142];

String decoded = utf8.decode(list);

print(decoded); // Hello😎

It’s fine to pass in a list of int values, but you’ll get an exception if any of them are larger than 8 bits so be careful of that.

Boolean logic

I’m assuming you know about the && , || , and ! Boolean logic operators that you use with the bool type:

// AND

print(true && true); // true

print(true && false); // false

print(false && false); // false // OR

print(true || true); // true

print(true || false); // true

print(false || false); // false // NOT

print(!true); // false

print(!false); // true

Well, there are also bitwise equivalents of these operators that work on binary numbers. If you think of false as 0 and true as 1 , you get similar results with the bitwise operators & , | , and ~ (plus an additional ^ XOR operator):

// AND

print(1 & 1); // 1

print(1 & 0); // 0

print(0 & 0); // 0 // OR

print(1 | 1); // 1

print(1 | 0); // 1

print(0 | 0); // 0 // XOR

print(1 ^ 1); // 0

print(1 ^ 0); // 1

print(0 ^ 0); // 0 // NOT

print(~1); // -2

print(~0); // -1

Well, the results are almost the same. The ~ bitwise NOT operator gave some strange numbers. We’ll come back to that in a bit, though you’ve probably guessed the reason already.

If you’d like to get a little more background in bitwise operations before we see some Dart examples, have a look at this video series:

Bitwise AND operator &

The bitwise & operator compares every pair of bits and gives a 1 in the resulting bit only if both of the input bits were 1 .

Here is an example:

final y = 0x4a; // 01001010

final x = 0x0f; // 00001111

final z = x & y; // 00001010

Why is that useful? Well, one useful thing to do is to make an AND bitmask. A bitmask is a way to filter out everything but the information you’re looking for.

You can see this in practice in the Flutter TextPainter source code:

static bool _isUtf16Surrogate(int value) {

return value & 0xF800 == 0xD800;

}

This static method checks a UTF-16 value to see if it is a surrogate pair. Surrogates are in the range 0xD800 – 0xDFFF . The value 0xF800 is the bitmask.

It’s easier to understand when you see it in boolean.

D800 1101 1000 0000 0000 (min)

DFFF 1101 1111 1111 1111 (max)

F800 1111 1000 0000 0000 (bitmask)

You notice that the first five bits of any surrogate are 11011 . The following eleven bits can be anything. So that’s where the bitmask comes in. It uses five 1 bits to match the pattern it’s looking for and eleven 0 bits to ignore the rest.

Recall that our friend Smiley 😎 is made of the surrogate pairs D83D and DE0E , so either of values should return true . Let’s try the second one, DE0E :

DE0E 1101 1110 0000 1110 (test value)

F800 1111 1000 0000 0000 (bitmask)

-------------------------

D800 1101 1000 0000 0000 (result of & operation)

The result is D800 so that should make the comparison true , meaning yes, DE0E is a surrogate value. You can test it out yourself:

bool _isUtf16Surrogate(int value) {

return value & 0xF800 == 0xD800;

} print(_isUtf16Surrogate(0xDE0E)); // true

Bitwise OR operator |

The bitwise | operator compares every pair of bits and has a 1 in the resulting bit if either (or both) of the input bits were 1 .

Here is an example:

final y = 0x4a; // 01001010

final x = 0x0f; // 00001111

final z = x | y; // 01001111

Why is that useful? Well, you can also use it as a bitmask to “turn on” certain bits while leaving other bits unaffected.

For example, colors are often packed into a single 32-bit integer, where the first 8 bits represent the alpha value (transparency), the next 8 bits are the red value, the next 8 bits the green value, and the final 8 bits the blue value.

So let’s say you have this semi-transparent purple color here:

If you want to keep the color but make it fully opaque, you could use an OR bitmask to do it. The hex value for that color is 802E028A :

alpha red green blue

-----------------------------------

80 2E 02 8A

10000000 00101110 00000010 10001010

Fully transparent is 0x00 and fully opaque is 0xFF . So you want to make a bitmask that will keep the red, green, and blue values the same, but make the alpha 0xFF . In this case we should expect FF2E028A for the result after the bitmask is applied.

10000000 00101110 00000010 10001010 (original)

11111111 00000000 00000000 00000000 (bitmask)

-----------------------------------

11111111 00101110 00000010 10001010 (result of | operation)

Since ORing anything with 1 makes 1 , this turned all of the alpha values to 1 . ORing anything with 0 doesn’t change the value, so the other values stayed the same.

Try it in Dart:

final original = 0x802E028A;

final bitmask = 0xFF000000;

final opaque = original | bitmask; print(opaque.toRadixString(16)); // ff2e028a

Got it!

Side note: If it bugs you that the hex string is in lowercase, you can always covert it to uppercase:

print('ff2e028a'.toUpperCase()); // FF2E028A

Bitwise XOR operator ^

The bitwise ^ operator compares every pair of bits and has a 1 in the resulting bit if the input bits are different.

Here’s an example:

final y = 0x4a; // 01001010

final x = 0x0f; // 00001111

final z = x ^ y; // 01000101

Why is that useful? In Dart you’ll often see the ^ operator used to create hash codes. For example, here is a Person class:

class Person {

final String name;

final int age; Person(this.name, this.age); @override

bool operator ==(Object other) {

return identical(this, other) ||

other is Person &&

runtimeType == other.runtimeType &&

name == other.name &&

age == other.age;

} @override

int get hashCode => name.hashCode ^ age.hashCode;

}

The last line with hashCode is the interesting part. Let’s reproduce that here:

final name = 'Bob';

final age = 97;

final hashCode = name.hashCode ^ age.hashCode; print(name.hashCode); // 124362681

print(age.hashCode); // 97

print(hashCode); // 124362712 print(name.hashCode.toRadixString(2).padLeft(32, '0'));

print(age.hashCode.toRadixString(2).padLeft(32, '0'));

print(hashCode.toRadixString(2).padLeft(32, '0'));

Here are the binary results:

00000111011010011001111110111001 (name hash code)

00000000000000000000000001100001 (age hash code)

--------------------------------

00000111011010011001111111011000 (result of ^ operation)

For hash codes you want them to be as distributed as possible so that you don’t get hash collisions. If you do & operations you get more 0 s, and if you do | operations you get more 1 s. Since using the ^ operator maintains the distribution of 0 s and 1 s, this makes it a good candidate for creating new hash codes from other ones.

Bitwise NOT operator ~

The ~ operator reverses the value of the bits. Earlier you saw these confusing results:

// NOT

print(~1); // -2

print(~0); // -1

This makes more sense if you look at the binary value. Here’s the 64-bit representation of 1:

0000000000000000000000000000000000000000000000000000000000000001

When you flip all of those bits (i.e., the result of ~1 ), you get:

1111111111111111111111111111111111111111111111111111111111111110

And if you watched the video about Two’s Complement, you’ll remember that this is how you express the number -2 in binary. It’s a similar story for ~0 . Problem solved.

By the way, here’s a joke for those of you who like Shakespeare:

2b|~2b

That is the question.

Bit shifting << >>

There’s one last topic to cover: bit shifting. You can shift bits to the left using the bitwise left shift operator << and to the right with the bitwise right shift operator >> .

You can learn more about bit shifting in this video:

Here’s an example in Dart of a left shift:

final value = 0x01; // 00000001

print(value << 5); // 00100000

All the bits were shifted left by 5. A binary value of 00100000 is 32. As an interesting fact, a simple way to multiply by two is shift left by one:

print(7 << 1); // 14 (decimal)

Here’s a right shift:

final value = 0x80; // 10000000

print(value >> 3); // 00010000

This is equivalent to saying 128 right shift 3 is 16.

Why is this important? Well, one place you can see it in Dart is to extract values from a packed integer. For example, the Color class source code has the following getter to extract the red value from a color value:

/// The red channel of this color in an 8 bit value.

int get red => (0x00ff0000 & value) >> 16;

This first uses an AND bitmask to find only the red part of the ARGB (alpha red green blue) value and then moves the result to the beginning. Here’s how that would look with our 802E028A purple color from before.

alpha red green blue

-----------------------------------

80 2E 02 8A

10000000 00101110 00000010 10001010 (original)

00000000 11111111 00000000 00000000 (bitmask)

-----------------------------------

00000000 00101110 00000000 00000000 (result of &)

-----------------------------------

00000000 00000000 00000000 00101110 (result of >> 16)

And here that is in Dart:

final purple = 0x802E028A;

final redBitmask = 0x00FF0000;

final masked = purple & redBitmask;

final redPart = masked >> 16;

print(redPart.toRadixString(16)); // 2e

Key points

You made it! Here are the key points from this article:

Binary data can be expressed as a list of integers.

Uint8List is an unsigned list of 8-bit integers and is more efficient than List<int> when working with large amounts of binary data.

is an unsigned list of 8-bit integers and is more efficient than when working with large amounts of binary data. You can use BytesBuilder to add binary data to a list.

to add binary data to a list. Byte data is backed by a BytesBuffer and you can get different views of the underlying bytes. For example, you can see the bytes in 8-bit, 16-bit, 32-bit, or 64-bit chunks, either as signed or unsigned integers.

and you can get different views of the underlying bytes. For example, you can see the bytes in 8-bit, 16-bit, 32-bit, or 64-bit chunks, either as signed or unsigned integers. With chunks larger than 8 bits you need to be aware of endianness, which is affected by the underlying machine or storage format. You can use ByteData to specify either big endian or little endian views.

to specify either big endian or little endian views. Prefix a number with 0x to write it in hexadecimal notation.

to write it in hexadecimal notation. Dart strings are lists of UTF-16 values and are known as code units.

Convert strings to UTF-8 by using the dart:convert library.

library. The bitwise logical operators are & , | , ^ , and ~ , and the shift operators are << and >> .

, , , and , and the shift operators are and . Bitwise operators and bitmasks allow you to access and manipulate individual bits.

Going on

A couple areas that deserve further study are byte streams and how to transform the data on those streams. For example, you can get a byte stream when reading or downloading a file. Rather than waiting for the whole file to arrive before working with it, you can start transforming the bytes to UTF-8 values (or something else) as you get them.

Remember to leave a comment or ask a question if anything is unclear.