If diamonds are made by replicators, that does not imply that they will be free, or as cheap as potatoes. In fact when you go from natural trees to diamond trees, the situation will change for the worse. Natural trees don't require much of an investment. Agricultural land is not expensive. Neither is fertilizer.
Agriculture is a fairly low-tech business.
Besides, an orange tree doesn't have to be designed. A diamond tree does. The nanofactory is going to require an investment, comparable to any other high-tech factory. A chip factory costs in the neighborhood of a billion dollars, and, as I argued in the previous section, a factory that makes use of atomic positioners will cost even more. Its products won't be free, and won't necessarily be cheap.