I'm also not sure where you're drawing the line here. Having a math degree myself, I'm absolutely fascinated by your distinctions. Keeping things concrete, while possibly useful for teaching calculation, is a hindrance for teaching mathematics, the primary idea of which is abstract generalization beyond the concrete. Set theory is the best starting point for learning the fundamental abstract concepts of mathematics. Well, I agree with you about set theory, but for an entirely different reason.
Sets are less abstract, more concrete-izable than numbers and operators.
You have a set of five blue blocks and a set of three red blocks. I find that addition is a naturally understood abstraction of set unions and subtraction follows from set intersection nicely. I firmly believe that children need to abstract a step at a time from the concrete, returning to the concrete to illustrate the abstract concepts and to test the abstractions. The concepts of mathematics are best illustrated using real objects. I have heard other mathematicians argue that math should begin, not with set theory, but with number theory.