SQL Server: Overview of different methods for change tracking
By: Shahar Gvirtz
Tags: Database, Programming, Software Development, Sql Server, Data Architecture

Crawling

Our first method is simple: perform periodic polling over the relevant tables, and ask “what’s changed since last time I did that polling”. Usually, this will involve tracking some columns, and always query for values that are bigger than the ones we already have (and keep some state of the most up-to-date value we’ve seen so far). For example, if we want to track the “Id” column (let’s say it’s auto-increment int), we’ll always SELECT for all the rows that have bigger Id than the one we saw in the previous run. After finishing processing the data, we’ll store the highest Id we’ve seen in this iteration, and next time use it as our new baseline.

What columns to track?

We have a few common options. If we’re interested in new rows only (and not updated ones), we’ll usually track one of the following:

Auto-increment numeric Id column : Many times it’ll be our primary key value, which is often an IDENTITY column. In that case, we’ll store (usually on a different state table, or state file) the last Id we saw, and SELECT for all the Id’s bigger than it.

This method, of course, works well when all we care about are only new rows. This method doesn’t help up getting information about updated rows.

: Many times it’ll be our primary key value, which is often an IDENTITY column. In that case, we’ll store (usually on a different state table, or state file) the last Id we saw, and SELECT for all the Id’s bigger than it. This method, of course, works well when all we care about are only new rows. This method doesn’t help up getting information about updated rows. “DateTimeCreated” column (with the date and time when the row was created): If we save the date and time when the row is created, we can use that value for our “crawling”. In that case, we’ll save in our state table the date and time of when we started selecting the new data from the table.

Of course, we need to think about time synchronization: We need to make sure there are no timezone differences between the time the application use, and the time our “crawler process” use. We also need to consider when the value in the table is created. For example, if the value is generated in the application (and not in the DB), we might miss rows due to race-condition (between the crawler process, and newly inserted values) or due to time difference between two clocks (it’s incorrect to assume synchronized clocks).

We can use this method safely only if the time is generated in the DB, or if we take some “safety gap” and also select rows we might already selected (and then filter those out, using some state we save).

How to choose between those options? If one of the columns (Id / RowDateCreated) is the key column of the table’s clustered index, then it’s better to use it. This will help us achieve better performance, as using the clustered index for range queries is the most efficient way (we save the key-lookup extra cost). If we use the clustered index, then selecting new rows will be a very cheap operation, with no performance impact at all.

Of course, many times we want to know about updates as well (and also deleted, if we use soft delete techniques, rather than actual deletes. I’ll discuss actual deletes later in this post).

The column I described earlier won’t help up knowing about updates, but then we can use a different set of columns:

DateTimeUpdated column, that includes both the date and the time the row was updated: It’s relevant only if we have this column, and the application updates it for each update. Personally, if there’s no such column, I wouldn’t create it and add a trigger to update it.

It’s important to remember that all the points from the earlier discussion about datetime values and how if not used correctly can make you miss data — are relevant here as well.

It’s relevant only if we have this column, and the application updates it for each update. Personally, if there’s no such column, I wouldn’t create it and add a trigger to update it. It’s important to remember that all the points from the earlier discussion about datetime values and how if not used correctly can make you miss data — are relevant here as well. Rowversion column: One of SQL Server datatypes is the rowversion (aka timestamp). The rowversion column is automatically updated for each change to the table, so if we updated a value (or added a new one), the rowversion value will be bigger than any rowversion data we’ve previously seen in that table. We can store the rowversion value in our state table, as a varbinary(8) column, and track values bigger than the biggest rowversion value we’ve seen so far.

Which option is better? Here it’s a bit trickier than detecting new rows. Every non clustered index we’ll create over any of those columns will suffer from a major fragmentation if our table is updated frequently. In a frequently updated table, this can cause a severe performance. Of course, it’s highly not recommended to use any of those columns as a clustered index key, which also means that our execution plan will also involve a key-lookup operation for each row, to get all the values from the clustered index (if we have non clustered index on this columns).

Temporal tables

Starting with SQL Server 2016, we have temporal tables. If the table we want to check for updates is a temporal table, we can combine both methods: Use the Id / DateTimeCreated column on the table to get new rows, and track updated rows by crawling the history table (after we’ll add an index on the SysEndTime column).

The rows we’ll receive from crawling the temporal tables will be the old versions (meaning, the values that were replaced during the UPDATE/DELETE). But, by using the primary key from the history table, we can query the main table and get the updated information.

Personally, I wouldn’t make a non-temporal table into a temporal table just for this, but if it’s already defined as a temporal table — it can be a nice trick to track updates/deletes as well.

Performance considerations

“Crawling” a table is basically constant polling from the SQL Server database. Meaning, to know there was a change, we need to run our query to get the latest changes.

Polling based methods, as a rule of thumb, are usually problematic. They usually involve higher performance costs. That’s because we have “busy wait” on our tables (because if there are no very frequent changes, most of the time our polling query will return no results).

But, because the “crawling” method is easy to implement, sometimes it’s chosen even when considering the cost involved.

Here are some rules of thumbs to help you reduce the performance impact of this polling:

Choose how often you run your tracking query carefully : If the service that consumes the data needs to get the updates, in your business case, only once every few hours — there’s no point in performing polling queries twice a second.

: If the service that consumes the data needs to get the updates, in your business case, only once every few hours — there’s no point in performing polling queries twice a second. Make sure you run only cheap queries: The performance cost of the polling query should be minimal, and it’s even more important if you run this query very frequently. Avoid queries that involve a lot of I/O operations, parallelism, spooling, etc.

The performance cost of the polling query should be minimal, and it’s even more important if you run this query very frequently. Avoid queries that involve a lot of I/O operations, parallelism, spooling, etc. Avoid complex queries: To make sure your crawling queries are cheap, as described above, avoid complex logic in those queries. The query should be basically something like SELET TOP X FROM Tbl WHERE ID > <number> (when you have at least a unique non-clustered index on the ID column, in that sample).

Anything more complicated than that, and you might encounter performance problems (if not now, then in the future). Even if the service you’re sending the data to needs to get aggregated data, this “crawling” query is not the right place to do that.

Even if your very-complex-sophisticated-query runs perfect and fast right now, don’t count on that. A slightly different execution plan in the future might change it dramatically and you’ll end up with heavy query running in a loop.

To make sure your crawling queries are cheap, as described above, avoid complex logic in those queries. The query should be basically something like SELET TOP X FROM Tbl WHERE ID > <number> (when you have at least a unique non-clustered index on the ID column, in that sample). Anything more complicated than that, and you might encounter performance problems (if not now, then in the future). Even if the service you’re sending the data to needs to get aggregated data, this “crawling” query is not the right place to do that. Even if your very-complex-sophisticated-query runs perfect and fast right now, don’t count on that. A slightly different execution plan in the future might change it dramatically and you’ll end up with heavy query running in a loop. Think what happens when you don’t have new data: If your query contains WHERE predicate, that select only data matching some criteria, but there are no new rows matching these criteria for hours/days/weeks — you might find yourself with a query that getting more complicated as the time passes, because you don’t update in your tracking table a new bigger ID that used as the starting point (for example). So, you have to think about what happens if there are no new rows, and how it affects the complexity of your queries.

If your query contains WHERE predicate, that select only data matching some criteria, but there are no new rows matching these criteria for hours/days/weeks — you might find yourself with a query that getting more complicated as the time passes, because you don’t update in your tracking table a new bigger ID that used as the starting point (for example). So, you have to think about what happens if there are no new rows, and how it affects the complexity of your queries. Think about moving the polling to the MSSQL-side: Imagine a pretty-standard scenario, where you have an application that performs the crawling. A naive implementation is to have a thread, with while(true) loop, that in every iteration SELECT’s new data from the MSSQL. If there are no new rows, it just begins another iteration of that loop.

The bad part of this implementation is that you have the worst of all worlds: you are doing polling queries in the MSSQL side, but also have busy-wait on the application side. A possible solution is to move some of the polling to be on the MSSQL side, like this: