Make the set screw a 4-40.

There are plenty of 4-40 set screws available, like
http://www.mcmaster.com/#93445a170/=tdk0uz, which specifies a hex key size of
0.05" (driver: http://www.mcmaster.com/#7400a11/=tdk1pc).

I can test this using 4-40 screws available from Radio Shack (catalog# 64-3011)
and meanwhile I have an order out to McMaster-Carr for those set screws and the
driver and 3/32" drill bits.