Brick is a building component that outperforms wood or plastic in many ways.  This is the reason a brick home is significantly more money than a home with PVC or wood siding.
In terms of recycling, brick is not as desirable.  Most all of the concrete and asphalt recycle yards do not want their end product contaminated with brick rubble.  Brick rubble turns into mush when wet so it cannot be recycled into road base.  It is a lot more expensive for us to dispose of brick rubble than asphalt or concrete.
The good news is that if you build with brick, then it could be someone several lifetimes from now who might be thinking of recycling.