Pin swift when using LKG=1

The current swift has a change that doesn't work with our other
LKG versions.