Deciding between Row- and Columnar-Stores | Why We Chose Both
By: Alexa Griffith
Tags: Bluecore, Data Engineering, Programming, Data, Software Engineering

Row oriented databases

Row-stores are considered “traditional” because they have been around longer than columnar-stores. Most row oriented databases are commonly known for OLTP (online transactional processing). This means that row-stores are most commonly known to perform well for a single transaction like inserting, updating, or deleting relatively small amounts of data. (3)

Writing one row at a time is easy for a row-store because it appends the whole row to a chunk of space in storage. In other words, the row oriented database is partitioned horizontally. Since each row occupies at least one chunk (a row can take up more than one chunk if it runs out of space), and a whole chunk of storage is read at a time, this makes it perfect for OLTP applications where a small number of records are queried at a time. (4)

row-stores save to storage in chunks

Using a row-store

Row-stores (ex: Postgres, MySQL) are beneficial when most/all of the values in the record (row) need to be accessed. Row oriented databases are also good for point lookups and index range scans. Indexing (creating a key from columns) based on your access patterns can optimize queries and prevent full table scans in a large row-store. If the value needed is in the index, you can pull it straight from there. Indexing is an important component of row-stores because while columnar-stores also have some indexing mechanisms to optimize full table scans, it is not as efficient for reducing seek time for individual record retrieval than an index on the appropriate columns. Note that creating many indices will create many copies of data, and a columnar-store is a better alternative (see When to Enable Indexing?). (5,6)

row-store concept overview

If only one field of the record is desired, then using a row-store becomes expensive since all the fields in each record will be read. Even data that isn’t needed for the query response will be read, assuming it isn’t indexed properly. Consequently, many seek operations are required to complete the query. For this reason, a columnar-store is favored when you have unpredictable access patterns, whereas known access patterns are well accommodated by a row-store. (7)

Column oriented databases

As more records in a database are accessed, the time to transfer data from disk to memory starts to outweigh the time it takes to seek the data. For this reason, columnar-stores are typically better for OLAP (online analytical processing) applications. Analytical applications often need aggregate data, where only a subset of a table’s attributes are needed. (8)

Column oriented databases are partitioned vertically — instead of storing the full row, the individual values are stored contiguously in storage by column. The advantage of a columnar-store is that partial reads are much more efficient because a lower volume of data is loaded due to reading only the relevant data instead of the whole record.

For example, if a chunk of storage can hold five values and the database has five columns (ex: one row has five values), one row will take up one chunk and be read together. If only one column value is needed for the query response, a columnar-store can read 5x as fast because you will read five column values in one chunk as opposed to one column value in the chunk containing the row. You also avoid reading the other column values that are irrelevant to the query response.

columnar-stores save columns to storage in chunks

Additionally, in column-stores compression is achieved more efficiently than in row-stores because columns have uniform types (ex: all strings or integers). These performance benefits apply to arbitrary access patterns, making them a good choice in the face of unpredictable queries. (8)

Using a columnar-store

example database

Columnar-stores (examples: RedShift, BigQuery) are good for computing trends and averages for trillions of rows and petabytes for data. (8)

Assuming this table continues for millions of rows, what if we wanted to know the sum of the amount spent on online purchases for company A? Well, for company A’s online purchases table, we would need to sum all of the online purchase values. Instead of going through each row and reading the email, type of purchase, and any other columns this table could have, we just need to access all of the values in the “amount” column.