Removed pointless cache bypass in Swift backend

* This was added by mistake in 2af7ad8 apparently

Change-Id: Ic4004428095953ce07e1e97da43748eba04f6639