[VBA] Performance Tuning — String
Tags: Hong Kong, Programming, Excel, Vba

I’m not a VBA expert, it’s just a story about how I optimized the VBA performance on my way, not a professional advice.

Again, 我唔係VBA expert, 我只係講我自己既嘗試. 目標要將一個原本要行成日既xlsm (CLS!?), tune到一個比較可以接受既時間.

I have to tweak a xlsm at work, the original xlsm which took several hours for the following processes:

Retrieving Market Data Generating SQL queries from the data Finally importing them into database.

(some of them approximately 500k rows)

Actually the speed of getting the market data is no way to boost, it’s pretty much depends on the connectivity status and the service provider. It took around 5–10 minutes to finish the retrieving process.

So that, I will look into the process of generating SQL queries and import to DB. I will sort it into 2 problems, one of them is from VBA, and the other one is from SQL Server.

份嘢主要就係做上面講嗰3樣嘢啦. Market Data 快慢取決於vendor同network, 大慨行5-10分鐘. 無乜得tune. 我睇過原本嗰VBA, 我會覺得有兩個問題導致佢咁慢, 一個來自VBA本身, 另一個來自SQL Server.

So, Let’s start ! Disable Excel Functions

First of all, I disabled some of the functions in Excel as below:

Sub OnStart() Application.ScreenUpdating = False

Application.DisplayStatusBar = False

Application.EnableEvents = False

Application.AskToUpdateLinks = False End Sub

一開始 Disable 咗上面啲 Functions, 保證唔會因為啲唔關事既嘢整慢咗佢

String concatenation

And then, I tweaked the sheet with 140k rows first. In the original xlsm, it took almost 5 hours to generate and run SQL queries. I have simplified some of the original code below, it was like:

攪咗張細啲既sheet先, generate 大慨14萬行sql , 佢原本大致係咁寫 (我簡化咗啲先post啦當然)

Dim Conn As New ADODB.Connection

Dim rowNo as Long: rowNo = 2 'Skip the header row

Dim sql as String Conn.Open "Provider=SQLOLEDB;Data Source=127.0.0.1;Initial Catalog=temp;User ID=temp;Password=temp;"

Conn.CommandTimeout = 0 With Worksheets("HongKong")

Do Until .Cells(rowNo, 1) = ""

sql = sql & "INSERT INTO [dbo].[tblHK]([id],[id1],[id2],[name], [description])VALUES( "& "'" _

& .Cells(rowNo, 1) & "'," _

& "'" & .Cells(rowNo, 2) & "'," _

& "'" & .Cells(rowNo, 3) & "'," _

& "'" & .Cells(rowNo, 4) & "'," _

& "'" & .Cells(rowNo, 5) & "'," _

& ");" rowNo= rowNo+ 1

Loop Conn.Execute sql

End With

As you can see, it will concatenate hundred thousand SQL queries into single variable, and then execute in SQL server. I think it was because the programmer wanted to avoid the hundred thousand round-trip to SQL server. But it was very slow to concatenate the string when the string grows very big. [20963 seconds = 5 hours]

我諗原本佢係諗住唔洗call 10九幾萬次個sql server哇, 但單係砌string 都攪5粒鐘. 所以首先試下砌下 string 睇下差幾遠

On the first attempt, I will add a bracket to concatenate all the string on the right side, it will significantly optimized the speed.

sql = sql & ("INSERT INTO ...;")

To improve the readability, I add an intermediate variable as below:

sql = "INSERT INTO ...;"

sqlQueries = sqlQueries & sql

After the change, it dramatically down to [1517 seconds = 25 minutes]

淨係加個variable就已經可以十分一時間攪掂...小改變大改善, 但可唔可以再快啲呢?

Can it be faster? I will try to change it to using JOIN, and see the result

...

Dim arr(140000) as String

...

With Worksheets("HongKong")

Do Until .Cells(rowNo, 1) = ""

sqlQuery = "INSERT INTO ..."

arr(rowNo) = sqlQuery

...

rowNo = rowNo +1

Loop

End With sqlQueries = Join(arr, ";")

It downs to 6 seconds only now !!

將啲string擺晒入個array, 再用JOIN, 就可以快到6秒攪掂, 由5個鐘變6秒...

點解咁快? 據講其實每次將個string 加上去嗰時, 佢要預留memory既位, string size + 1. 個1係放 the size of the string.

(PS: 所以 LEN function 會行得快, 因為其實唔洗count).

咁每次要加一個string 上去, 佢都要再搵一個新同連續夠位放既address in memory, 所以當個string 越嚟越大, 佢就越花時間搵. 但用JOIN, 因為本身array已經係fixed size, 佢就唔洗咁多嘢諗. 以上既嘢係我既理解, 但我搵過, 又未搵到MSDN 既supporting document, 所以如果有錯 或者有其他 idea, 請指正

當然段SQL query 都有嘢執, 或者改咗做 BULK INSERT. 但依篇主要講String, 就唔深究住了.

Result:

Original : 20963 sec

Intermediate Variable : 1517 sec

using JOIN : 6 sec

Of course, there are alternative ways are to change the SQL to multiple insert, like VALUES (…), (…), (…); and BULK INSERT, which means save the records into a new .csv, and then run BULK INSERT to import the file. But, it’s a little bit related to the handling in SQL server, I’m not going to discuss in this story.

Finally, I think it still has room for optimization, if you have a better idea about it, please let me know. And please correct me if I were wrong.