Introduction to Streams and Buffers
By: Uday Hiwarale
Tags: Go Programming Language, Golang, Golang Tutorial, Go Programming, Programming

Reading from a Data Source

A data-source is a data container (in memory or file storage) that can stream data. This container implements basic interfaces provided by the io package and exposes some methods to read data from.

io.Reader

To be able to read data from a source as a stream, the source must implement io.Reader interface. The io.Reader interface declares the basic Read method. Any type that implements the Read method is a type of Reader .

type Reader interface {

Read(p []byte) (n int, err error)

}

The Read method accepts a slice of bytes p and Read will copy first len(p) bytes from the source into the packet p . When next Read call is made, next len(p) bytes from the source is copied into p until there is no more data is left to copy from the source, in which case the Read method will return io.EOF (end-of-file) error.

The n in the above definition is the number of bytes read by each Read method call. In some cases, actual source data length can be less than the len(p) or the data left in the source can be less than len(p) , hence n and err can be very helpful to identify such cases.

When Read is reading the last packet of data, Read it can return io.EOF error so that we do not make Read call again and again. When there is some data still left to read, it should return a nil error. When there is absolutely no data left to read, Read call should return n=0 and err=io.EOF .

Let’s first break down the implementation of the Reader interface. In the example above, we have a struct type MyStringData which holds a string str and a read index readIndex . The readIndex field will be used to hold the index of the character in str from which the Read method should return the next packet of data.

This struct type implements Read method defined by the Reader interface. The receiver of this method is a pointer of MyStringData type, because this method will modify the readIndex field, hence we need the actual MyStringData object instead of just a copy of it.

Inside Read method definition, we create a slice of bytes from the source which is myStringData.src since we are working with bytes than string characters. The readIndex tracks from where to start reading the source. Since the array index in Go starts at zero (as so for the slices), at an ideal end condition of the program, readIndex should be the length of the source.

When the Read() call is made and the readIndex is greater than or equal to the source length, we simply return the EOF error with n set to 0 which indicates that the 0 bytes were read and it is an end of the file has reached.

If that’s not the case, we simply get the limit to which data should be read starting from the readIndex . Since p packet is the data holder, we simply add readIndex and len(p) to get the nextReadLimit .

However, we also need to consider if nextReadLimit exceeds the length of the source and if it does, we need to set it to a safe limit which is the length of the source and also specify that it is the end of the file condition.

Next, we read from readIndex up until nextReadLimit and copy to these bytes nextBytes into the holder packet p . As we have learned from the slices lesson, we do not need a pointer of a slice as an argument to mutate it.

The number of actual bytes read n can be less than len(p) , hence n is always the length of nextBytes . Before returning anything from Read method, we need to change the value of readIndex so that next Read call will start reading from the last cursor position.

In the main function, we have created a src struct of type MyStringData and initialized the src field value with 20 character length string. We have set a buffer-packet of size 3 . This means, for every Read call, 3 bytes will be read.

Then we have started an infinite for loop that reads from src until an error has occurred. Since io.EOF is not a serious error, we should have a separate handling mechanism for it. This program yields the following result.

🌶 go run go-streams.go

3 bytes read, data: Hel

3 bytes read, data: lo

3 bytes read, data: Ama

3 bytes read, data: zin

3 bytes read, data: g W

3 bytes read, data: orl

2 bytes read, data: d!

--end-of-file--

Notice that the last Read call actually received only 2 bytes, because at that point, only 2 bytes were left to read. This is the reason we are using p[:n] slice extract operator to read up until the actual bytes read from the packet because packet could contain bytes from the previous Read call.

💡 There is a serious issue with the example above. Since were are using only ASCII characters in this example, this example works just fine. But if we have Unicode characters which take more than 1 code units (bytes), then we can have code units of the same character split between two Read calls and the program wouldn’t be able to print those characters successfully.

strings.NewReader

If you are thinking, all that for a drop of blood, I mean, for just a string, then do not panic. As I said, Go provides streaming capabilities over some built-in types. To get an object from a string that implements the Read method, we can use strings.NewReader function.

This function returns a pointer to strings.Reader struct that implements the Read method defined by the io.Reader interface. Hence it is considered as a io.Reader object and we can stream data from it using the Read method.

In the above example, though the implementation remains the same, we now create the Reader object from strings.NewReader function. This program yields the following result.

🌶 go run go-streams.go

3 bytes read, data: Hel

3 bytes read, data: lo

3 bytes read, data: Ama

3 bytes read, data: zin

3 bytes read, data: g W

3 bytes read, data: orl

2 bytes read, data: d!

0 bytes read, data:

--end-of-file--

The last Read call might surprise you. The last Read call returned n=0 and err=EOF . In our implementation, we returned the EOF error with the last packet (when we’re sure that no further Read call was possible).

ioutil.ReadAll

If you want to read everything from a source which implements io.Reader interface, then you can use ioutil.ReadAll method.

func ReadAll(src io.Reader) ([]byte, error)

The ReadAll method will read from src until src returns an EOF error or some other uneventful error. The ReadAll method will only return an error if the underlying source src returns this uneventful error (other than EOF ).

🌶 go run go-streams.go

Read data of length 20 : Hello Amazing World!

io.ReadFull

If you are interested in exactly x number of bytes from the source and somehow, if x number of bytes could not be read, get an error instead, io.ReadFull is the function for you.

func ReadFull(src Reader, buf []byte) (n int, err error)

The ReadFull function will read from the source src exactly len(buf) bytes. In an ideal condition, number of bytes read n will be len(buf) . If n is less than len(buf) because src returned EOF , then ReadFull will return an io.ErrUnexpectedEOF error.

If ReadFull reads 0 bytes from src (when n=0 ), the error returned by the ReadFull is io.EOF . In a special condition, if an error is returned by src after len(buf) bytes were read, that error is dropped by ReadFull .

In the above example, we have a source of length 20 and a buffer buf of length 14 . We can guess that first ReadFull call will be successful because 14 bytes can be read from the source src .

Next call, however, won’t be successful because only 6 bytes are left to read and ReadFull will return io.ErrUnexpectedEOF error because src will return EOF before all 14 bytes are read.

For any subsequent calls, ReadFull will always return io.EOF error because the source src has been exhausted and no more data can be read. Hence, the program above yields the following result.

🌶 go run go-streams.go

Bytes read: 14, value: Hello Amazing , err: <nil>

Bytes read: 6, value: World!, err: unexpected EOF

Bytes read: 0, value: , err: EOF

💡 If you are interested in x number bytes of data from a source like we saw above but you are ok with at least y number of bytes, then you can use io.ReadAtLeast function. You can also implement the io.ReaderAt interface to provide capability to read from any index point of the source.

io.LimitReader

Many at times, we need to set a cap on a data source. The io.LimitReader takes a Reader object r and returns a Reader object with a cap of n bytes.

func LimitReader(r Reader, n int64) Reader

The LimitReader function returns a Reader object that abstracts source r . This returned Reader object will keep reading from r until n bytes are read and it will return io.EOF error once n bytes are completely read.

In the example above, we have the main data source mainSrc of 20 bytes length. But we need to only read 10 bytes, hence we have created another source src from mainSrc using io.LimitReader .

The packet size for the Read call is 3 which means 4 Read() calls will be successful until EOF is reached. Let’s see the result.