Begin building the bridge from decoder to encoder