Made some kind of progress... I think...
Changed the pieces to SKSpriteNodes instead of SKShapeNodes to increase performance.